Similarity calculation method, device, equipment and storage medium based on word features

By using a similarity calculation method based on word features in an intelligent question-and-answer system, the accuracy and scalability of text similarity calculation in the prior art are solved, and the similarity calculation in specific business scenarios is improved Accuracy and adaptability.

CN111259126BActive Publication Date: 2024-11-19CHINA PING AN LIFE INSURANCE CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202010042471.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-01-15
Publication Date
2024-11-19
Estimated Expiration
2040-01-15

AI Technical Summary

Technical Problem

In the existing intelligent question and answer system, semantic representation method based on deep learning model requires a large amount of data sample size and slow iteration, while single-level literal matching method cannot effectively reflect the contribution degree of words in specific business scenarios, affecting text similarity. accuracy of degree calculation.

Method used

Using a similarity calculation method based on word characteristics, by obtaining the original problem text, determining the target application scenario and word segmentation standards, extracting word characteristics, calculating the similarity between forward and reverse texts, and performing feature fusion to generate similarity matching scores. To reflect the semantic inclusion relationship of words in specific business scenarios.

Benefits of technology

It improves the accuracy of text similarity calculation in specific business scenarios, prioritizes matching of required word types, and adapts to the rapid expansion needs of intelligent question-and-answer systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111259126B_ABST
    Figure CN111259126B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of artificial intelligence, and discloses a similarity calculation method, device, equipment and storage medium based on word features, which are used to improve the calculation accuracy of text similarity values ​​in specific business scenarios. The method of the present invention includes: obtaining an original question text; determining a target application scenario according to the original question text and a preset application scenario, and obtaining a target word segmentation standard corresponding to the target application scenario and a plurality of semantically similar similar question texts; selecting any one of the multiple semantically similar similar question texts as a candidate question text, and extracting the word features of the original question text and the word features of the candidate question text according to the target word segmentation standard; obtaining forward text similarity and reverse text similarity; generating a similarity matching score; determining the candidate similarity value with the largest value as the target similarity value, and selecting the candidate question text corresponding to the target similarity value as the standard question text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of similarity matching, and in particular to a similarity calculation method, device, equipment and storage medium based on word features. Background Art

[0002] In traditional customer service systems or training systems, a large amount of manpower and resources are often required to respond to business requests, and the professionalism and proficiency of the staff are required to be high. The operating costs remain high and the human resource retention rate is low. Therefore, in the process of intelligent transformation, the demand for intelligent question and answer systems is imminent.

[0003] At present, the mainstream way to build intelligent question-answering systems in the industry is based on the retrieval method, that is, recalling the questions most similar to the user's questions from the knowledge base of the question-answering system, among which text similarity calculation is the core module in the retrieval recall. Existing text similarity calculation methods mainly include deep learning model methods and single-level literal matching calculations (such as keyword matching, edit distance, Jaccard similarity, etc.). Among them, the semantic representation method based on the deep learning model has a large demand for data sample size, and the iteration of the new question library corpus is slow and not easy to expand. The single-level literal matching method essentially sets the weight of each token (word / character) to be the same, which cannot effectively reflect the contribution of words at different levels in specific business scenarios, thereby affecting the measurement of text similarity in specific business scenarios. Summary of the invention

[0004] The present invention provides a similarity calculation method, device, equipment and storage medium based on word features, which are used to reflect the word types that need to be matched first in specific business scenarios, embody the semantic inclusion relationship of texts, and improve the calculation accuracy of text similarity values ​​in specific business scenarios.

[0005] A first aspect of an embodiment of the present invention provides a similarity calculation method based on word features, comprising: obtaining an original question text, the original question text being used to indicate a search for an answer corresponding to the original question text; determining a target application scenario according to the original question text and a preset application scenario, and obtaining a target word segmentation standard corresponding to the target application scenario and a plurality of semantically similar similar question texts, the preset application scenario comprising a plurality of pre-set candidate scenarios; selecting any one of the plurality of semantically similar similar question texts as a candidate question text, and extracting word features of the original question text and word features of the candidate question text according to the target word segmentation standard; performing calculations according to the word features of the original question text and the word features of the candidate question text respectively to obtain a forward text similarity and a reverse text similarity; performing feature fusion on the forward text similarity and the reverse text similarity to generate a similarity matching score, the similarity matching score being used to indicate the degree of similarity between the original question text and the candidate question text; comparing the candidate similarity values ​​corresponding to the plurality of candidate question texts, determining the candidate similarity value with the largest value as the target similarity value, and selecting the candidate question text corresponding to the target similarity value as the standard question text.

[0006] Optionally, in a first implementation method of the first aspect of the embodiment of the present invention, the target application scenario is determined according to the original question text and a preset application scenario, and a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics are obtained, and the preset application scenario includes a plurality of preset candidate scenarios, including: selecting any one application scenario from the preset application scenarios as the target application scenario according to the original question text, and the preset application scenario includes a plurality of preset application scenarios; obtaining the target word segmentation standard corresponding to the target application scenario; and searching for similar question texts with similar semantics to the original question text under the target application scenario.

[0007] Optionally, in a second implementation method of the first aspect of the embodiment of the present invention, selecting any one similar question text from the multiple semantically similar similar question texts as a candidate question text, and extracting word features of the original question text and word features of the candidate question text according to the target word segmentation standard, includes: selecting any one similar question text from the multiple semantically similar similar question texts as a candidate question text; performing word segmentation and named entity recognition on the original question text and the candidate question text respectively based on the target word segmentation standard to obtain word features of the original question text and word features of the candidate question text.

[0008] Optionally, in a third implementation manner of the first aspect of the embodiment of the present invention, the original question text and the candidate question text are respectively segmented and named entity recognized based on the target segmentation standard to obtain word features of the original question text and word features of the candidate question text, including: segmenting the original question text based on the target segmentation standard to obtain a word segmentation result of the original question text; segmenting the candidate question text based on the target segmentation standard to obtain a word segmentation result of the candidate question text; performing named entity recognition on the word segmentation result of the original question text and the word segmentation result of the candidate question text, respectively, to obtain word features of the original question text and word features of the candidate question text, the word features of the original question text include marked original words and corresponding original word parts of speech, and the word features of the candidate question text include marked candidate words and corresponding candidate word parts of speech.

[0009] Optionally, in a fourth implementation manner of the first aspect of the embodiment of the present invention, the performing word segmentation and named entity recognition on the original question text and the candidate question text respectively based on the target word segmentation standard to obtain word features of the original question text and word features of the candidate question text includes: performing word segmentation on the original question text based on the target word segmentation standard to obtain a word segmentation result of the original question text; obtaining a preset word segmentation result of the candidate question text, wherein the preset word segmentation result is a result of performing offline word segmentation on the candidate question text in advance according to the target word segmentation standard; performing named entity recognition on the word segmentation result of the original question text and the preset word segmentation result of the candidate question text respectively to obtain word features of the original question text and word features of the candidate question text, wherein the word features of the original question text include labeled original words and corresponding original word parts of speech, and the word features of the candidate question text include labeled candidate words and corresponding candidate word parts of speech.

[0010] Optionally, in a fifth implementation of the first aspect of the embodiment of the present invention, the calculating according to the word features of the original question text and the word features of the candidate question text respectively to obtain the forward text similarity and the reverse text similarity includes:

[0011] The original question text is determined as the benchmark question text, the candidate question text is determined as the matching question text, and the positive text similarity is calculated based on the preset matching formula. The preset matching formula is A represents the benchmark question text, B represents the matching question text, and L A Indicates the number of word tokens in the benchmark question text A, w A,i Represents the normalized weights of all levels of tokens in the benchmark text A.A,i Indicates the token and token corresponding to the subscript of the benchmark question text B,j Indicates the token corresponding to the subscript of the matching question text, and jaccard indicates the similarity coefficient of the two tokens. The candidate question text is determined as the benchmark question text, the original question text is determined as the matching question text, and the reverse text similarity is calculated based on a preset matching formula.

[0012] Optionally, in a sixth implementation manner of the first aspect of the embodiment of the present invention, the forward text similarity and the reverse text similarity are feature fused to generate a similarity matching score, and the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text, including: fusing the forward text similarity and the reverse text similarity through a preset formula, the preset formula is: score = w1*score(forward)+w2*score(reverse)+b, wherein b is a constant, and w1 and w2 are weight constants; calculating a similarity matching score score, and the similarity matching score indicates the degree of similarity between the original question text and the candidate question text.

[0013] A second aspect of an embodiment of the present invention provides a similarity calculation device based on word features, comprising: an acquisition unit, used to acquire an original question text, wherein the original question text is used to indicate a search for an answer corresponding to the original question text; a determination unit, used to determine a target application scenario according to the original question text and a preset application scenario, and to acquire a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics, wherein the preset application scenario includes a plurality of pre-set candidate scenarios; a selection and extraction unit, used to select any one of the plurality of similar question texts with similar semantics as a candidate question text, and to extract the word features of the original question text according to the target word segmentation standard. The invention relates to a method for preparing a standard question text comprising: a first method for preparing a standard question text and a second method for preparing a standard question text; a second method for preparing a standard question text; a third method for preparing a standard question text; a fourth method for preparing a standard question text; a fourth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text; a fifth method for preparing a standard question text;

[0014] Optionally, in a first implementation method of the second aspect of the embodiment of the present invention, the determination unit is specifically used to: select any one application scenario from preset application scenarios as a target application scenario based on the original question text, the preset application scenario including multiple pre-set application scenarios; obtain a target word segmentation standard corresponding to the target application scenario; and search for similar question texts with similar semantics to the original question text under the target application scenario.

[0015] Optionally, in a second implementation method of the second aspect of the embodiment of the present invention, the selection and extraction unit includes: a selection module, used to select any one similar question text from the multiple semantically similar similar question texts as a candidate question text; a word segmentation recognition module, used to perform word segmentation and named entity recognition on the original question text and the candidate question text respectively based on the target word segmentation standard to obtain the word features of the original question text and the word features of the candidate question text.

[0016] Optionally, in a third implementation manner of the second aspect of the embodiment of the present invention, the word segmentation recognition module is specifically used to: perform word segmentation on the original question text based on the target word segmentation standard to obtain a word segmentation result of the original question text; perform word segmentation on the candidate question text based on the target word segmentation standard to obtain a word segmentation result of the candidate question text; perform named entity recognition on the word segmentation result of the original question text and the word segmentation result of the candidate question text, respectively, to obtain word features of the original question text and word features of the candidate question text, the word features of the original question text include marked original words and corresponding original word parts of speech, and the word features of the candidate question text include marked candidate words and corresponding candidate word parts of speech.

[0017] Optionally, in a fourth implementation manner of the second aspect of the embodiment of the present invention, word segmentation recognition is specifically used for: performing word segmentation on the original question text based on the target word segmentation standard to obtain a word segmentation result of the original question text; obtaining a preset word segmentation result of the candidate question text, wherein the preset word segmentation result is a result of offline word segmentation of the candidate question text in advance according to the target word segmentation standard; performing named entity recognition on the word segmentation result of the original question text and the preset word segmentation result of the candidate question text, respectively, to obtain word features of the original question text and word features of the candidate question text, wherein the word features of the original question text include labeled original words and corresponding original word parts of speech, and the word features of the candidate question text include labeled candidate words and corresponding candidate word parts of speech.

[0018] Optionally, in a fifth implementation of the second aspect of the embodiment of the present invention, the calculation unit is specifically used to: determine the original question text as the benchmark question text, determine the candidate question text as the matching question text, and calculate the positive text similarity based on a preset matching formula, and the preset matching formula is A represents the benchmark question text, B represents the matching question text, and L A Indicates the number of word tokens in the benchmark question text A, w A,i Represents the normalized weights of all levels of tokens in the benchmark text A. A,i Indicates the token and token corresponding to the subscript of the benchmark question text B,j Indicates the token corresponding to the subscript of the matching question text, and jaccard indicates the similarity coefficient of the two tokens. The candidate question text is determined as the benchmark question text, the original question text is determined as the matching question text, and the reverse text similarity is calculated based on a preset matching formula.

[0019] Optionally, in a sixth implementation manner of the second aspect of the embodiment of the present invention, the generation unit is specifically used to: fuse the forward text similarity and the reverse text similarity through a preset formula, the preset formula being: score=w1*score(forward)+w2*score(reverse)+b, wherein b is a constant, and w1 and w2 are weight constants; and calculate a similarity matching score score, wherein the similarity matching score indicates the degree of similarity between the original question text and the candidate question text.

[0020] A third aspect of an embodiment of the present invention provides a similarity calculation device based on word features, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the similarity calculation method based on word features described in any of the above embodiments is implemented.

[0021] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the similarity calculation method based on word features described in any of the above embodiments are implemented.

[0022] In the technical solution provided by the embodiment of the present invention, an original question text is obtained, and the original question text is used to indicate the search for the answer corresponding to the original question text; a target application scenario is determined according to the original question text and a preset application scenario, and a target word segmentation standard corresponding to the target application scenario and a plurality of semantically similar similar question texts are obtained, wherein the preset application scenario includes a plurality of pre-set candidate scenarios; any one similar question text is selected from the plurality of semantically similar similar question texts as a candidate question text, and word features of the original question text and word features of the candidate question text are extracted according to the target word segmentation standard; forward text similarity and reverse text similarity are calculated according to the word features of the original question text and the word features of the candidate question text respectively; the forward text similarity and the reverse text similarity are feature fused to generate a similarity matching score, and the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text; the candidate similarity values ​​corresponding to the plurality of candidate question texts are compared, the candidate similarity value with the largest value is determined as the target similarity value, and the candidate question text corresponding to the target similarity value is selected as the standard question text. The embodiment of the present invention adopts a forward matching method based on user questions and a reverse matching method based on standard questions to respectively calculate the forward text similarity and the reverse text similarity, and fuses the forward text similarity and the reverse text similarity to obtain a final similarity calculation value, which reflects the word types that need to be matched preferentially in specific business scenarios, embodies the text semantic inclusion relationship, and improves the calculation accuracy of the text similarity value in specific business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of an embodiment of a method for calculating similarity based on word features in an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of another embodiment of a method for calculating similarity based on word features in an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of an embodiment of a similarity calculation device based on word features in an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of another embodiment of a similarity calculation device based on word features in an embodiment of the present invention;

[0027] Figure 5 The figure is a schematic diagram of an embodiment of a similarity calculation device based on word features in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention provides a similarity calculation method, device, equipment and storage medium based on word features, which are used to reflect the word types that need to be matched first in specific business scenarios, embody the semantic inclusion relationship of texts, and improve the calculation accuracy of text similarity values ​​in specific business scenarios.

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] See also Figure 1 , a flowchart of a similarity calculation method based on word features provided by an embodiment of the present invention specifically includes:

[0032] 101. Obtain the original question text, where the original question text is used to indicate the answer to be found corresponding to the original question text.

[0033] The server obtains the original question text, and the original question text is used to indicate to find the answer corresponding to the original question text.

[0034] It should be noted that for an original question text, there may be multiple corresponding answers. For example, the original question text is: "What subjects are included in the physical examination?" The corresponding answers may include one or more of the following answers: "Otolaryngology", "Routine blood test", "Five hepatitis B tests", "and routine urine test", "electrocardiogram" or "B-ultrasound", etc., and the specific answers are not limited here.

[0035] It is understandable that the execution subject of the present invention may be a similarity calculation device based on word features, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0036] 102. Determine a target application scenario based on the original question text and a preset application scenario, and obtain a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics. The preset application scenario includes a plurality of preset candidate scenarios.

[0037] Specifically, (1) selecting any one application scenario from preset application scenarios as a target application scenario according to the original question text, where the preset application scenarios include multiple preset application scenarios;

[0038] (2) Obtaining the target word segmentation standard corresponding to the target application scenario;

[0039] Different business scenarios have different word segmentation standards. Words in different business scenarios can be divided into multiple different levels, and the word weights of each level can be set. For example, in the actual application of Ping An Life Insurance's intelligent question-answering system, the word segmentation standards are divided into six categories: insurance product entities, disease name entities, location name entities, occupation name entities, service operation keywords (such as "insurance", "claims"), and other words. The weights of words in different categories are different.

[0040] (3) Search for similar question texts that are semantically similar to the original question text in the target application scenario.

[0041] Among them, there will be multiple similar question texts in the target application scenario. For example, in the intelligent question and answer system for life insurance, it can include: "Ping An Fu insurance", "How to insure Ping An Fu" and "How to pay Ping An Fu". If the original question text is "I want to insure Ping An Fu", then "Ping An Fu insurance" and "How to insure Ping An Fu" can both be identified as similar standard question texts.

[0042] It should be noted that the same question may be expressed in different ways. For example, the original question text is: "What are the key points of pre-job training?", and similar question texts with similar semantics to the original question text may be: "Key content of pre-job training" or "What are the key points of pre-job training?". For another example, the original question text is: "Insurance for gastroenteritis", and similar question texts with similar semantics to the original question text may be: "Can gastroenteritis be insured?" or "Insurance for acute gastroenteritis" or "Can acute gastroenteritis be insured?".

[0043] 103. Select any one similar question text from multiple similar question texts with similar semantics as a candidate question text, and extract the word features of the original question text and the word features of the candidate question text according to the target word segmentation standard.

[0044] Specifically: the server selects any one similar question text from multiple semantically similar similar question texts as a candidate question text; the server performs word segmentation and named entity recognition on the original question text and the candidate question text respectively based on the target word segmentation standard to obtain the word features of the original question text and the word features of the candidate question text.

[0045] Among them, the server performs word segmentation and named entity recognition on the original question text and the candidate question text respectively based on the target word segmentation standard to obtain the word features of the original question text and the word features of the candidate question text, which may include: the server performs word segmentation on the original question text based on the target word segmentation standard to obtain the word segmentation result of the original question text; the server performs word segmentation on the candidate question text based on the target word segmentation standard to obtain the word segmentation result of the candidate question text; the server performs named entity recognition on the word segmentation result of the original question text and the word segmentation result of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

[0046] For example, for the original question text "insurance for gastroenteritis", "Can insurance for gastroenteritis be purchased?" is selected as the candidate question text. It should be noted that entity recognition, keyword recognition, etc. can be in the form of NER models or dictionaries, which are not limited here.

[0047] Alternatively, it may also include: the server performs word segmentation on the original question text based on the target word segmentation standard to obtain the word segmentation result of the original question text; the server obtains the preset word segmentation result of the candidate question text, wherein the preset word segmentation result is the result of offline word segmentation of the candidate question text in advance according to the target word segmentation standard; the server performs named entity recognition on the word segmentation result of the original question text and the preset word segmentation result of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, wherein the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

[0048] For example, the original question text is: "I want to insure Ping An Fu", and the candidate question text is: "Ping An Fu insured". The original question text and the candidate question text are preprocessed. In practice, some words can be removed as pause words. After the question text is preprocessed, the preset entity recognition model or keyword recognition model is used to obtain the word features of each level of the original question text and the candidate question text. For example, the word features of the original question text are: (I, other words)(want, other words)(insured, keywords)(Ping An Fu, insurance entity words). The word features of the candidate question text are: (Ping An Fu, insurance entity words)(insured, keywords).

[0049] 104. Calculate the word features of the original question text and the word features of the candidate question text respectively to obtain the forward text similarity and the reverse text similarity.

[0050] The server determines the original question text as the benchmark question text, determines the candidate question text as the matching question text, and calculates the positive text similarity based on the preset matching formula. The preset matching formula is A represents the benchmark question text, B represents the matching question text, and L A Indicates the number of word tokens in the benchmark question text A, w A,i Represents the normalized weights of all levels of tokens in the benchmark text A. A,i Indicates the token and token corresponding to the subscript of the benchmark question text B,j Indicates the token corresponding to the subscript of the matching question text, and jaccard indicates the similarity coefficient of the two tokens. The server determines the candidate question text as the benchmark question text, determines the original question text as the matching question text, and calculates the reverse text similarity based on a preset matching formula.

[0051] For example, the original question text is: "I want to insure Ping An Fu", and the candidate question text is: "Ping An Fu insured"; the word features of the original question text are: (I, other words)(want, other words)(insured, keywords)(Ping An Fu, insurance entity words)(insured, keywords); the word features of the candidate question text are: (Ping An Fu, insurance entity words)(insured, keywords). Assuming that the weight of entity words (including) is 3, the weight of keywords is 2, and the weight of other words is 1, then the forward matching results for the original question text A "I want to insure Ping An Fu" and the candidate question text B "Ping An Fu insured" are: Score (forward) = 1 / (1+1+2+3)*max(jaccard(I, Ping An Fu), jaccard(I, insured))+1 / (1+1+2+3)*max(jaccard(want, Ping An Fu), jaccard(want, insured))+2 / (1+1+2+3)*max(jaccard(insured, Ping An Fu), jaccard(insured, insured))+3 / (1+1+2+3)*max(jaccard(Ping An Fu, Ping An Fu), jaccard(Ping An Fu, insured))=1 / 8*0+1 / 8*0+2 / 8*1+3 / 8*1=5 / 8. The reverse matching result is: Score(reverse) = 3 / (3+2)*max(jaccard(Ping An Fu, me), jaccard(Ping An Fu, want), jaccard(Ping An Fu, insurance), jaccard(Ping An Fu, Ping An Fu))+2 / (3+2)*max(jaccard(insurance, me), jaccard(insurance, want), jaccard(insurance, insurance), jaccard(insurance, Ping An Fu))=3 / 5+2 / 5=1.

[0052] 105. The forward text similarity and the reverse text similarity are feature fused to generate a similarity matching score, which is used to indicate the similarity between the original question text and the candidate question text.

[0053] The server performs feature fusion of the forward text similarity and the reverse text similarity to generate a similarity matching score, which is used to indicate the degree of similarity between the original question text and the candidate question text. Specifically, the server fuses the forward text similarity and the reverse text similarity through a preset formula, which is: score = w1*score (forward) + w2*score (reverse) + b, where b is a constant, and w1 and w2 are weight constants; the server calculates a similarity matching score score, which indicates the degree of similarity between the original question text and the candidate question text.

[0054] For example, the original question text (user question) is: "Can gastric cancer be insured with Ping An Fu?", and the candidate question text 1 contained in the question library under the corresponding business scenario is: "Can gastric cancer be insured with Ping An Fu?", and the candidate question text 2 is: "Can gastric cancer be insured with Ping An Fu and Ai Man Score?". Assuming that the weights of feature fusion are: w1=0.6, w2=0.4, b=0, then the similarity matching score of candidate question text 1 is calculated to be: score1=0.6*(8 / 9)+0.4*(8 / 10)=0.853, and the similarity matching score of candidate question text 2 is: score2=0.6*(8 / 9)+0.4*(8 / 14)=0.762. Obviously, "Can stomach cancer be insured with Ping An Fu" and "Can stomach cancer be insured with Ping An Fu" are semantically equivalent, while "Can stomach cancer be insured with Ping An Fu" and "Can stomach cancer be insured with Ping An Fu and Ai Manfen" are semantically not equivalent; therefore, for the life insurance intelligent question-answering system, the similarity matching score of candidate question text 1 should be greater than the similarity matching score of candidate question text 2. If there is only a positive match, the scores of the two candidate question texts are the same, both 8 / 9, which is unreasonable, because candidate question text 2 semantically contains the original question text and is not equivalent to the original question text. At the same time, forward matching and reverse matching are considered, and the text similarity matching algorithm is improved, which optimizes the problem of text semantic inclusion to a certain extent.

[0055] 106. Compare the candidate similarity values ​​corresponding to the multiple candidate question texts, determine the candidate similarity value with the largest value as the target similarity value, and select the candidate question text corresponding to the target similarity value as the standard question text.

[0056] The server compares the candidate similarity values ​​corresponding to the multiple candidate question texts, determines the candidate similarity value with the largest value as the target similarity value, and selects the candidate question text corresponding to the target similarity value as the standard question text.

[0057] For each original question text, there will be N candidate question texts. The server calculates the similarity between each candidate question text and the original question text, sorts them according to the similarity value, finds the candidate question text with the highest similarity, and determines it as the standard question text.

[0058] The embodiment of the present invention adopts a forward matching method based on user questions and a reverse matching method based on standard questions to respectively calculate the forward text similarity and the reverse text similarity, and fuses the forward text similarity and the reverse text similarity to obtain a final similarity calculation value, which reflects the word types that need to be matched preferentially in specific business scenarios, embodies the text semantic inclusion relationship, and improves the calculation accuracy of the text similarity value in specific business scenarios.

[0059] See also Figure 2, another flow chart of a similarity calculation method based on word features provided by an embodiment of the present invention, specifically comprising:

[0060] 201. Obtain the original question text, where the original question text is used to indicate the search for the answer corresponding to the original question text.

[0061] The server obtains the original question text, and the original question text is used to indicate to find the answer corresponding to the original question text.

[0062] It should be noted that for an original question text, there may be multiple corresponding answers. For example, the original question text is: "What subjects are included in the physical examination?" The corresponding answers may include one or more of the following answers: "Otolaryngology", "Routine blood test", "Five hepatitis B tests", "and routine urine test", "electrocardiogram" or "B-ultrasound", etc., and the specific answers are not limited here.

[0063] It is understandable that the execution subject of the present invention may be a similarity calculation device based on word features, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0064] 202. Determine a target application scenario based on the original question text and a preset application scenario, and obtain a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics. The preset application scenario includes a plurality of preset candidate scenarios.

[0065] Specifically, (1) selecting any one application scenario from preset application scenarios as a target application scenario according to the original question text, where the preset application scenarios include multiple preset application scenarios;

[0066] (2) Obtaining the target word segmentation standard corresponding to the target application scenario;

[0067] Different business scenarios have different word segmentation standards. Words in different business scenarios can be divided into multiple different levels, and the word weights of each level can be set. For example, in the actual application of Ping An Life Insurance's intelligent question-answering system, the word segmentation standards are divided into six categories: insurance product entities, disease name entities, location name entities, occupation name entities, service operation keywords (such as "insurance", "claims"), and other words. The weights of words in different categories are different.

[0068] (3) Search for similar question texts that are semantically similar to the original question text in the target application scenario.

[0069] Among them, there will be multiple similar question texts in the target application scenario. For example, in the intelligent question and answer system for life insurance, it can include: "Ping An Fu insurance", "How to insure Ping An Fu" and "How to pay Ping An Fu". If the original question text is "I want to insure Ping An Fu", then "Ping An Fu insurance" and "How to insure Ping An Fu" can both be identified as similar standard question texts.

[0070] It should be noted that the same question may be expressed in different ways. For example, the original question text is: "What are the key points of pre-job training?", and similar question texts with similar semantics to the original question text may be: "Key content of pre-job training" or "What are the key points of pre-job training?". For another example, the original question text is: "Insurance for gastroenteritis", and similar question texts with similar semantics to the original question text may be: "Can gastroenteritis be insured?" or "Insurance for acute gastroenteritis" or "Can acute gastroenteritis be insured?".

[0071] 203. Select any one of multiple similar question texts with similar semantics as a candidate question text.

[0072] The server selects any one similar question text from multiple similar question texts with similar semantics as a candidate question text.

[0073] For example, for the original question text "Gastroenteritis insurance", similar question texts with similar semantics to the original question text are: "Can gastroenteritis be insured?" or "Acute gastroenteritis insurance" or "Can I buy insurance for acute gastroenteritis?", select "Can gastroenteritis be insured" as the candidate question text.

[0074] 204. Based on the target word segmentation standard, the original question text and the candidate question text are respectively segmented and named entity recognized to obtain the word features of the original question text and the word features of the candidate question text.

[0075] Specifically, the server segments the original question text based on the target segmentation standard to obtain the segmentation result of the original question text; the server segments the candidate question text based on the target segmentation standard to obtain the segmentation result of the candidate question text; the server performs named entity recognition on the segmentation result of the original question text and the segmentation result of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

[0076] Alternatively, the server performs word segmentation on the original question text based on the target word segmentation standard to obtain the word segmentation result of the original question text; the server obtains the preset word segmentation result of the candidate question text, wherein the preset word segmentation result is the result of offline word segmentation of the candidate question text in advance according to the target word segmentation standard; the server performs named entity recognition on the word segmentation result of the original question text and the preset word segmentation result of the candidate question text, respectively, to obtain the word features of the original question text and the word features of the candidate question text, wherein the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

[0077] For example, the original question text is: "I want to insure Ping An Fu", and the candidate question text is: "Ping An Fu insured". The original question text and the candidate question text are preprocessed. In practice, some words can be removed as pause words. After the question text is preprocessed, the preset entity recognition model or keyword recognition model is used to obtain the word features of each level of the original question text and the candidate question text. For example, the word features of the original question text are: (I, other words)(want, other words)(insured, keywords)(Ping An Fu, insurance entity words). The word features of the candidate question text are: (Ping An Fu, insurance entity words)(insured, keywords).

[0078] 205. Calculate the word features of the original question text and the word features of the candidate question text respectively to obtain the forward text similarity and the reverse text similarity.

[0079] The server determines the original question text as the benchmark question text, determines the candidate question text as the matching question text, and calculates the positive text similarity based on the preset matching formula. The preset matching formula is A represents the benchmark question text, B represents the matching question text, and L A Indicates the number of word tokens in the benchmark question text A, w A,i Represents the normalized weights of all levels of tokens in the benchmark text A. A,i Indicates the token and token corresponding to the subscript of the benchmark question text B,j Indicates the token corresponding to the subscript of the matching question text, and jaccard indicates the similarity coefficient of the two tokens. The server determines the candidate question text as the benchmark question text, determines the original question text as the matching question text, and calculates the reverse text similarity based on a preset matching formula.

[0080] For example, the original question text is: "I want to insure Ping An Fu", and the candidate question text is: "Ping An Fu insured"; the word features of the original question text are: (I, other words)(want, other words)(insured, keywords)(Ping An Fu, insurance entity words)(insured, keywords); the word features of the candidate question text are: (Ping An Fu, insurance entity words)(insured, keywords). Assuming that the weight of entity words (including) is 3, the weight of keywords is 2, and the weight of other words is 1, then the forward matching results for the original question text A "I want to insure Ping An Fu" and the candidate question text B "Ping An Fu insured" are: Score (forward) = 1 / (1+1+2+3)*max(jaccard(I, Ping An Fu), jaccard(I, insured))+1 / (1+1+2+3)*max(jaccard(want, Ping An Fu), jaccard(want, insured))+2 / (1+1+2+3)*max(jaccard(insured, Ping An Fu), jaccard(insured, insured))+3 / (1+1+2+3)*max(jaccard(Ping An Fu, Ping An Fu), jaccard(Ping An Fu, insured))=1 / 8*0+1 / 8*0+2 / 8*1+3 / 8*1=5 / 8. The reverse matching result is: Score(reverse) = 3 / (3+2)*max(jaccard(Ping An Fu, me), jaccard(Ping An Fu, want), jaccard(Ping An Fu, insurance), jaccard(Ping An Fu, Ping An Fu))+2 / (3+2)*max(jaccard(insurance, me), jaccard(insurance, want), jaccard(insurance, insurance), jaccard(insurance, Ping An Fu))=3 / 5+2 / 5=1.

[0081] 206. The forward text similarity and the reverse text similarity are feature fused to generate a similarity matching score, which is used to indicate the similarity between the original question text and the candidate question text.

[0082] The server performs feature fusion of the forward text similarity and the reverse text similarity to generate a similarity matching score, which is used to indicate the degree of similarity between the original question text and the candidate question text. Specifically, the server fuses the forward text similarity and the reverse text similarity through a preset formula, which is: score = w1*score (forward) + w2*score (reverse) + b, where b is a constant, and w1 and w2 are weight constants; the server calculates a similarity matching score score, which indicates the degree of similarity between the original question text and the candidate question text.

[0083] For example, the original question text (user question) is: "Can gastric cancer be insured with Ping An Fu?", and the candidate question text 1 contained in the question library under the corresponding business scenario is: "Can gastric cancer be insured with Ping An Fu?", and the candidate question text 2 is: "Can gastric cancer be insured with Ping An Fu and Ai Man Score?". Assuming that the weights of feature fusion are: w1=0.6, w2=0.4, b=0, then the similarity matching score of candidate question text 1 is calculated to be: score1=0.6*(8 / 9)+0.4*(8 / 10)=0.853, and the similarity matching score of candidate question text 2 is: score2=0.6*(8 / 9)+0.4*(8 / 14)=0.762. Obviously, "Can stomach cancer be insured with Ping An Fu" and "Can stomach cancer be insured with Ping An Fu" are semantically equivalent, while "Can stomach cancer be insured with Ping An Fu" and "Can stomach cancer be insured with Ping An Fu and Ai Manfen" are semantically not equivalent; therefore, for the life insurance intelligent question-answering system, the similarity matching score of candidate question text 1 should be greater than the similarity matching score of candidate question text 2. If there is only a positive match, the scores of the two candidate question texts are the same, both 8 / 9, which is unreasonable, because candidate question text 2 semantically contains the original question text and is not equivalent to the original question text. At the same time, forward matching and reverse matching are considered, and the text similarity matching algorithm is improved, which optimizes the problem of text semantic inclusion to a certain extent.

[0084] 207. Compare the candidate similarity values ​​corresponding to the multiple candidate question texts, determine the candidate similarity value with the largest value as the target similarity value, and select the candidate question text corresponding to the target similarity value as the standard question text.

[0085] The server compares the candidate similarity values ​​corresponding to the multiple candidate question texts, determines the candidate similarity value with the largest value as the target similarity value, and selects the candidate question text corresponding to the target similarity value as the standard question text.

[0086] For each original question text, there will be N candidate question texts. The server calculates the similarity between each candidate question text and the original question text, sorts them according to the similarity value, finds the candidate question text with the highest similarity, and determines it as the standard question text.

[0087] The embodiment of the present invention adopts a forward matching method based on user questions and a reverse matching method based on standard questions to respectively calculate the forward text similarity and the reverse text similarity, and fuses the forward text similarity and the reverse text similarity to obtain a final similarity calculation value, which reflects the word types that need to be matched preferentially in specific business scenarios, embodies the text semantic inclusion relationship, and improves the calculation accuracy of the text similarity value in specific business scenarios.

[0088] The above describes the similarity calculation method based on word features in the embodiment of the present invention. The following describes the similarity calculation device based on word features in the embodiment of the present invention. Figure 3 , an embodiment of a similarity calculation device based on word features in an embodiment of the present invention includes:

[0089] An acquisition unit 301 is used to acquire an original question text, where the original question text is used to indicate to search for an answer corresponding to the original question text;

[0090] A determination unit 302 is used to determine a target application scenario according to the original question text and a preset application scenario, and obtain a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics, wherein the preset application scenario includes a plurality of preset candidate scenarios;

[0091] A selection and extraction unit 303 is used to select any one of the multiple similar question texts with similar semantics as a candidate question text, and extract the word features of the original question text and the word features of the candidate question text according to the target word segmentation standard;

[0092] A calculation unit 304 is used to calculate the word features of the original question text and the word features of the candidate question text to obtain the forward text similarity and the reverse text similarity;

[0093] A generating unit 305, configured to perform feature fusion on the forward text similarity and the reverse text similarity to generate a similarity matching score, wherein the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text;

[0094] The comparison and selection unit 306 is used to compare the candidate similarity values ​​corresponding to the multiple candidate question texts, determine the candidate similarity value with the largest value as the target similarity value, and select the candidate question text corresponding to the target similarity value as the standard question text.

[0095] The embodiment of the present invention adopts a forward matching method based on user questions and a reverse matching method based on standard questions to respectively calculate the forward text similarity and the reverse text similarity, and fuses the forward text similarity and the reverse text similarity to obtain a final similarity calculation value, which reflects the word types that need to be matched preferentially in specific business scenarios, embodies the text semantic inclusion relationship, and improves the calculation accuracy of the text similarity value in specific business scenarios.

[0096] See also Figure 4 Another embodiment of the similarity calculation device based on word features in the embodiment of the present invention includes:

[0097] An acquisition unit 301 is used to acquire an original question text, where the original question text is used to indicate to search for an answer corresponding to the original question text;

[0098] A determination unit 302 is used to determine a target application scenario according to the original question text and a preset application scenario, and obtain a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics, wherein the preset application scenario includes a plurality of preset candidate scenarios;

[0099] A selection and extraction unit 303 is used to select any one of the multiple similar question texts with similar semantics as a candidate question text, and extract the word features of the original question text and the word features of the candidate question text according to the target word segmentation standard;

[0100] A calculation unit 304 is used to calculate the word features of the original question text and the word features of the candidate question text to obtain the forward text similarity and the reverse text similarity;

[0101] A generating unit 305, configured to perform feature fusion on the forward text similarity and the reverse text similarity to generate a similarity matching score, wherein the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text;

[0102] The comparison and selection unit 306 is used to compare the candidate similarity values ​​corresponding to the multiple candidate question texts, determine the candidate similarity value with the largest value as the target similarity value, and select the candidate question text corresponding to the target similarity value as the standard question text.

[0103] Optionally, the determining unit 302 is specifically configured to:

[0104] According to the original question text, any one application scenario is selected from the preset application scenarios as a target application scenario, wherein the preset application scenarios include a plurality of pre-set application scenarios; a target word segmentation standard corresponding to the target application scenario is obtained; and a similar question text with similar semantics to the original question text is searched under the target application scenario.

[0105] Optionally, the selecting and extracting unit 303 includes:

[0106] A selection module 3031 is used to select any one of the multiple similar question texts with similar semantics as a candidate question text;

[0107] The word segmentation recognition module 3032 is used to perform word segmentation and named entity recognition on the original question text and the candidate question text respectively based on the target word segmentation standard to obtain word features of the original question text and word features of the candidate question text.

[0108] Optionally, the word segmentation recognition module 3032 is specifically used for:

[0109] The original question text is segmented based on the target segmentation standard to obtain the segmentation result of the original question text; the candidate question text is segmented based on the target segmentation standard to obtain the segmentation result of the candidate question text; named entity recognition is performed on the segmentation result of the original question text and the segmentation result of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, wherein the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

[0110] Optionally, the word segmentation recognition module 3032 is further used for:

[0111] The original question text is segmented based on the target segmentation standard to obtain the segmentation result of the original question text; the preset segmentation result of the candidate question text is obtained, wherein the preset segmentation result is the result of offline segmentation of the candidate question text in advance according to the target segmentation standard; named entity recognition is performed on the segmentation result of the original question text and the preset segmentation result of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, wherein the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

[0112] Optionally, the calculation unit 304 is specifically configured to:

[0113] The original question text is determined as the benchmark question text, the candidate question text is determined as the matching question text, and the positive text similarity is calculated based on the preset matching formula. The preset matching formula is: A represents the benchmark question text, B represents the matching question text, and L A Indicates the number of word tokens in the benchmark question text A, w A,i Represents the normalized weights of all levels of tokens in the benchmark text A. A,i Indicates the token and token corresponding to the subscript of the benchmark question text B,jIndicates the token corresponding to the subscript of the matching question text, and jaccard indicates the similarity coefficient of the two tokens. The candidate question text is determined as the benchmark question text, the original question text is determined as the matching question text, and the reverse text similarity is calculated based on a preset matching formula.

[0114] Optionally, the generating unit 305 is specifically configured to:

[0115] The forward text similarity and the reverse text similarity are fused through a preset formula, the preset formula is: score = w1*score(forward)+w2*score(reverse)+b, wherein b is a constant, w1 and w2 are weight constants; the similarity matching score is calculated, and the similarity matching score indicates the degree of similarity between the original question text and the candidate question text.

[0116] The embodiment of the present invention adopts a forward matching method based on user questions and a reverse matching method based on standard questions to respectively calculate the forward text similarity and the reverse text similarity, and fuses the forward text similarity and the reverse text similarity to obtain a final similarity calculation value, which reflects the word types that need to be matched preferentially in specific business scenarios, embodies the text semantic inclusion relationship, and improves the calculation accuracy of the text similarity value in specific business scenarios.

[0117] above Figure 3 to Figure 4 The similarity calculation device based on word features in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The similarity calculation device based on word features in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0118] Figure 5 1 is a schematic diagram of a similarity calculation device based on word features provided by an embodiment of the present invention. The similarity calculation device 500 based on word features may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 501 (for example, one or more processors) and a memory 509, and one or more storage media 508 (for example, one or more mass storage devices) storing application programs 507 or data 506. Among them, the memory 509 and the storage medium 508 may be short-term storage or permanent storage. The program stored in the storage medium 508 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the similarity calculation device based on word features. Furthermore, the processor 501 may be configured to communicate with the storage medium 508 to execute a series of instruction operations in the storage medium 508 on the similarity calculation device 500 based on word features.

[0119] The similarity calculation device 500 based on word features may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems 505, such as Windows Serve, Mac OSX, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the similarity calculation device based on word features shown in the figure does not constitute a limitation on the similarity calculation device based on word features, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 501 can perform the functions of the acquisition unit 301, the determination unit 302, the selection and extraction unit 303, the calculation unit 304, the generation unit 305, and the comparison and selection unit 306 in the above-mentioned embodiment.

[0120] Combine the following Figure 5 The following is a detailed introduction to the various components of the similarity calculation device based on word features:

[0121] The processor 501 is the control center of the similarity calculation device based on word features, and can process according to the similarity calculation method based on word features. The processor 501 uses various interfaces and lines to connect the various parts of the similarity calculation device based on word features, and executes various functions and processes data of the similarity calculation device based on word features by running or executing software programs and / or modules stored in the memory 509, and calling data stored in the memory 509, thereby improving the recall rate of keywords, improving the comprehensive score of each keyword, and improving the accuracy of keyword extraction. The storage medium 508 and the memory 509 are both carriers for storing data. In the embodiment of the present invention, the storage medium 508 can refer to an internal memory with a small storage capacity but a fast speed, and the memory 509 can be an external memory with a large storage capacity but a slow storage speed.

[0122] The memory 509 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the similarity calculation device 500 based on word features by running the software programs and modules stored in the memory 509. The memory 509 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as calculating the word features of the original question text and the word features of the candidate question text respectively to obtain the forward text similarity and the reverse text similarity), etc.; the data storage area can store data created according to the use of the similarity calculation device based on word features (such as similarity matching scores, etc.), etc. In addition, the memory 509 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. The similarity calculation method program based on word features and the received data stream provided in the embodiment of the present invention are stored in the memory, and when needed, the processor 501 calls from the memory 509.

[0123] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A similarity calculation method based on word features, characterized in that: include: Obtaining an original question text, wherein the original question text is used to indicate a search for an answer corresponding to the original question text; Determine a target application scenario according to the original question text and a preset application scenario, and obtain a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics, wherein the preset application scenario includes a plurality of preset candidate scenarios; Selecting any one of the multiple semantically similar question texts as a candidate question text, and extracting word features of the original question text and word features of the candidate question text according to the target word segmentation standard; Calculating respectively according to the word features of the original question text and the word features of the candidate question text to obtain the forward text similarity and the reverse text similarity; Performing feature fusion on the forward text similarity and the reverse text similarity to generate a similarity matching score, wherein the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text; Compare the candidate similarity values ​​corresponding to the multiple candidate question texts, determine the candidate similarity value with the largest value as the target similarity value, and select the candidate question text corresponding to the target similarity value as the standard question text; The method of calculating the word features of the original question text and the word features of the candidate question text respectively to obtain the forward text similarity and the reverse text similarity includes: determining the original question text as the benchmark question text, determining the candidate question text as the matching question text, and calculating the forward text similarity based on a preset matching formula, wherein the preset matching formula is , where A represents the benchmark question text and B represents the matching question text. Indicates the number of word tokens in the benchmark question text A, represents the normalized weights of all levels of tokens in the benchmark text A. Indicates the subscript corresponding to the benchmark question text , Indicates the subscript corresponding to the matching question text , Indicates two The similarity coefficient of ; The candidate question text is determined as the benchmark question text, the original question text is determined as the matching question text, and the reverse text similarity is calculated based on a preset matching formula.

2. The similarity calculation method based on word features according to claim 1 is characterized in that: The target application scenario is determined according to the original question text and the preset application scenario, and a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics are obtained, wherein the preset application scenario includes a plurality of preset candidate scenarios, including: Selecting any one application scenario from preset application scenarios as a target application scenario according to the original question text, wherein the preset application scenario includes a plurality of preset application scenarios; Obtaining a target word segmentation standard corresponding to the target application scenario; Searching for similar question texts with semantic similarity to the original question text in the target application scenario.

3. The similarity calculation method based on word features according to claim 1 is characterized in that: The step of selecting any one of the multiple similar question texts with similar semantics as a candidate question text, and extracting the word features of the original question text and the word features of the candidate question text according to the target word segmentation standard includes: Select any one of the multiple similar question texts with similar semantics as a candidate question text; Based on the target word segmentation standard, word segmentation and named entity recognition are performed on the original question text and the candidate question text respectively to obtain word features of the original question text and word features of the candidate question text.

4. The similarity calculation method based on word features according to claim 3 is characterized in that: The step of performing word segmentation and named entity recognition on the original question text and the candidate question text based on the target word segmentation standard to obtain word features of the original question text and word features of the candidate question text includes: Segmenting the original question text based on the target segmentation standard to obtain a segmentation result of the original question text; Segmenting the candidate question text based on the target segmentation standard to obtain a segmentation result of the candidate question text; Named entity recognition is performed on the word segmentation results of the original question text and the word segmentation results of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, wherein the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

5. The similarity calculation method based on word features according to claim 3 is characterized in that: The step of performing word segmentation and named entity recognition on the original question text and the candidate question text based on the target word segmentation standard to obtain word features of the original question text and word features of the candidate question text includes: Segmenting the original question text based on the target segmentation standard to obtain a segmentation result of the original question text; Obtaining a preset word segmentation result of the candidate question text, wherein the preset word segmentation result is a result of pre-performing offline word segmentation on the candidate question text according to the target word segmentation standard; Named entity recognition is performed on the word segmentation results of the original question text and the preset word segmentation results of the candidate question text respectively to obtain the word features of the original question text and the word features of the candidate question text, wherein the word features of the original question text include the marked original words and the corresponding original word parts of speech, and the word features of the candidate question text include the marked candidate words and the corresponding candidate word parts of speech.

6. The similarity calculation method based on word features according to any one of claims 1 to 5, characterized in that: The step of fusing the forward text similarity and the reverse text similarity to generate a similarity matching score, wherein the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text, includes: The forward text similarity and the reverse text similarity are fused through a preset formula, the preset formula is: ,in, is the positive text similarity, is the reverse text similarity, b is a constant, w1 and w2 are weight constants; Calculate the similarity matching score , the similarity matching score indicates the similarity between the original question text and the candidate question text.

7. A similarity calculation device based on word features, characterized in that: include: An acquisition unit, used to acquire an original question text, wherein the original question text is used to indicate to search for an answer corresponding to the original question text; A determination unit, configured to determine a target application scenario according to the original question text and a preset application scenario, and obtain a target word segmentation standard corresponding to the target application scenario and a plurality of similar question texts with similar semantics, wherein the preset application scenario includes a plurality of preset candidate scenarios; A selection and extraction unit is used to select any one of the multiple similar question texts with similar semantics as a candidate question text, and extract the word features of the original question text and the word features of the candidate question text according to the target word segmentation standard; A calculation unit, used to calculate respectively according to the word features of the original question text and the word features of the candidate question text to obtain a forward text similarity and a reverse text similarity; A generating unit, used for performing feature fusion of the forward text similarity and the reverse text similarity to generate a similarity matching score, wherein the similarity matching score is used to indicate the degree of similarity between the original question text and the candidate question text; A comparison and selection unit, configured to compare the candidate similarity values ​​corresponding to the plurality of candidate question texts, determine the candidate similarity value with the largest value as the target similarity value, and select the candidate question text corresponding to the target similarity value as the standard question text; The calculation unit is specifically used to: determine the original question text as the benchmark question text, determine the candidate question text as the matching question text, and calculate the positive text similarity based on the preset matching formula. The preset matching formula is , where A represents the benchmark question text and B represents the matching question text. Indicates the number of word tokens in the benchmark question text A, represents the normalized weights of all levels of tokens in the benchmark text A. Indicates the subscript corresponding to the benchmark question text , Indicates the subscript corresponding to the matching question text , Indicates two The similarity coefficient of ; The candidate question text is determined as the benchmark question text, the original question text is determined as the matching question text, and the reverse text similarity is calculated based on a preset matching formula.

8. A similarity calculation device based on word features, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for calculating similarity based on word features as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the similarity calculation method based on word features as described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Short text similarity calculating method

    CN106484664A

  • Text similarity calculation method, text similarity calculation device and server

    CN108052509A

  • Dialog method and device based on artificial intelligence and computer readable medium

    CN108170749A