Method, device, equipment and storage medium for selecting error correction phrase candidates
By calculating the primary weight value and the secondary scoring and sorting model, the high accuracy candidates were selected, which solved the problem of low efficiency of existing error correction techniques and improved the speed and accuracy of error correction processing.
Patent Information
- Application Number
- CN202010164836.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-11
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-03-11
AI Technical Summary
The existing error correction technology is inefficient in handling user typos or language conversion errors, resulting in low accuracy of filtered candidates.
By reading the candidates for the phrase to be corrected, calculating the primary weight value and sorting it, obtaining the primary alternative candidates, calling the secondary scoring sorting model, filtering the candidates with the highest secondary weight value as the target candidate, and replacing it.
It improves error correction efficiency, reduces the number of irrelevant candidates entering the secondary order, and improves error correction processing speed and accuracy.
Smart Images

Figure CN111444315B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a method, device, equipment and computer-readable storage medium for screening error correction phrase candidates. Background Art
[0002] With the development of technology, artificial intelligence is becoming more and more common. For example, in human-computer dialogue scenarios, the machine needs to accurately understand the user's intention through the conversation content with the user. It is usually necessary to correct the user's wrong words or errors in the text caused by language conversion in the user's conversation content. Existing error correction technologies mainly need to go through processes such as error detection, candidate recall, and candidate sorting.
[0003] In the prior art, the existing error correction model mainly corrects wrong words typed by users or errors in proper nouns caused by language conversion. A single model is needed to screen and match all candidate items for the error correction phrase. This method often requires all candidate items to enter the secondary candidates, resulting in low accuracy of the screened candidates and low error correction efficiency. Summary of the invention
[0004] The main purpose of the present invention is to provide a method, device, equipment and computer-readable storage medium for screening error correction phrase candidates, aiming to solve the technical problem of low efficiency of existing error correction technology.
[0005] To achieve the above object, the present invention provides a method for screening error correction phrase candidate items, the method for screening error correction phrase candidate items comprising the following steps:
[0006] Read multiple candidate items corresponding to the phrase to be corrected;
[0007] Based on the attribute value of each candidate item, the primary weight value of each candidate item is calculated and sorted to determine the first sorting result of the candidate items, wherein the attribute value of the candidate items includes word frequency, edit distance and pinyin jaccard distance;
[0008] Based on the first sorting result, obtaining a plurality of first candidate items corresponding to the phrase to be corrected;
[0009] Calling a preset secondary scoring and sorting model, respectively obtaining the secondary weight values of the first candidate items and sorting them, to obtain a second sorting result of the first candidate items;
[0010] Based on the second sorting result, obtaining a plurality of second candidate items corresponding to the phrase to be corrected;
[0011] The second alternative candidate items with the highest secondary weight value among the second alternative candidate items are screened, and the corresponding second alternative candidate items are marked as target candidate items.
[0012] Optionally, before reading multiple candidate items corresponding to the phrase to be corrected, the method further includes:
[0013] Acquire corpus data and use the corpus data as a training sample set;
[0014] Based on the training sample set, extracting first parameter features of the training sample set, wherein the first parameter features include word frequency change features, word segmentation change features and language model features;
[0015] Based on the first parameter feature of the training sample set, the training sample set is trained using the XGboost algorithm to construct a secondary scoring and ranking model.
[0016] Optionally, the calculating the primary weight value of each candidate item and ranking the candidate items based on the attribute value of each candidate item, and determining the first ranking result of the candidate items includes:
[0017] Respectively obtain the attribute value of each candidate item, wherein the attribute value of the candidate item includes word frequency, edit distance and pinyin jaccard distance;
[0018] Based on the word frequency, edit distance and pinyin jaccard distance of the candidate items, respectively calculate the primary weight values of the candidate items;
[0019] The following formula is used to calculate the primary weight value of the candidate item:
[0020] M=log 10 (T)-PQ;
[0021] Among them, M represents the primary weight value of the candidate, T represents the word frequency of the corresponding candidate, P represents the edit distance of the corresponding candidate, and Q represents the pinyin jaccard distance;
[0022] Based on the primary weight values of the candidate items, the candidate items are sorted according to the weight values to obtain a first sorting result.
[0023] Optionally, calling a preset secondary scoring and sorting model, respectively obtaining the secondary weight values of the first candidate items and sorting them, and obtaining a second sorting result of the first candidate items includes:
[0024] Extracting second parameter features of the first candidate items respectively, wherein the second parameter features include word frequency change features, word segmentation change features and language model features;
[0025] Based on the second parameter feature, calling a preset secondary scoring and ranking model to respectively calculate the secondary weight values of the first candidate items;
[0026] According to the secondary weight values of the first alternative candidate items, the first alternative candidate items are sorted in a preset order to obtain a second sorting result of the first alternative candidate items.
[0027] Optionally, obtaining a plurality of second candidate items corresponding to the phrase to be corrected based on the second sorting result includes:
[0028] Based on the second sorting result, screening the first candidate items;
[0029] If there is a first candidate item whose secondary weight value is less than the preset threshold, the first candidate item is eliminated;
[0030] If there is a first alternative candidate item whose secondary weight value is greater than the preset threshold, the first alternative candidate item is retained;
[0031] The first alternative candidate items retained after screening are marked as second alternative candidates.
[0032] Optionally, after the step of screening the second candidate items with the highest secondary weight value among the second candidate items and marking the corresponding second candidate items as target candidate items, the method further includes:
[0033] The target candidate item is used as a replacement item to replace the phrase to be corrected.
[0034] Furthermore, the present invention also provides a device for screening error correction phrase candidate items, the device for screening error correction phrase candidate items comprising:
[0035] A reading module, used for reading multiple candidate items corresponding to the phrase to be corrected;
[0036] A calculation module, used to calculate the primary weight value of each candidate item based on the attribute value of each candidate item and sort them, so as to determine a first sorting result of the candidate items;
[0037] A first acquisition module, configured to acquire a plurality of first candidate items corresponding to the phrase to be corrected based on the first sorting result;
[0038] A secondary calculation module, used to call a preset secondary scoring and ranking model, respectively calculate the secondary weight values of the first candidate items and sort them, and obtain a second ranking result of the first candidate items;
[0039] A second acquisition module, configured to acquire a plurality of second candidate items corresponding to the phrase to be corrected based on the second sorting result;
[0040] The marking module is used to select the second alternative candidate items with the highest secondary weight value among the second alternative candidate items, and mark the corresponding second alternative candidate items as target candidate items.
[0041] Optionally, the device for screening error-correcting phrase candidates further comprises:
[0042] A third acquisition module is used to acquire corpus data and use the corpus data as a training sample set;
[0043] An extraction module, configured to extract a first parameter feature of the training sample set based on the training sample set;
[0044] A construction module is used to train the training sample set using an XGboost algorithm based on the first parameter feature of the training sample set to construct a secondary scoring and ranking model.
[0045] Optionally, the calculation module is specifically used for:
[0046] Acquire the attribute value of each candidate item respectively, wherein the attribute value of the candidate item includes word frequency, edit distance and pinyin jaccard distance; calculate the primary weight value of the candidate item respectively based on the word frequency, edit distance and pinyin jaccard distance of the candidate item; sort the candidate items according to the weight value based on the primary weight value of the candidate item to obtain a first sorting result;
[0047] The following formula is used to calculate the primary weight value of the candidate item:
[0048] M=log 10 (T)-PQ;
[0049] Among them, M represents the primary weight value of the candidate, T represents the word frequency of the corresponding candidate, P represents the edit distance of the corresponding candidate, and Q represents the pinyin jaccard distance.
[0050] Optionally, the secondary calculation module is specifically used for:
[0051] Extract second parameter features of the first alternative candidate items respectively, wherein the second parameter features include word frequency change features, word segmentation change features and language model features; based on the second parameter features, call a preset secondary scoring and sorting model to respectively calculate the secondary weight values of the first alternative candidate items; according to the secondary weight values of the first alternative candidate items, sort the first alternative candidate items in a preset order to obtain a second sorting result of the first alternative candidate items.
[0052] Optionally, the second acquisition module is further configured to:
[0053] Based on the second sorting result, the first alternative candidate items are screened; if there is a first alternative candidate item whose secondary weight value is less than the preset threshold, the first alternative candidate item is eliminated; if there is a first alternative candidate item whose secondary weight value is greater than the preset threshold, the first alternative candidate item is retained; the first alternative candidate item retained after screening is marked as the second alternative candidate item.
[0054] Furthermore, the device for screening error correction phrase candidates further comprises:
[0055] A replacement module is used to use the target candidate item as a replacement item to replace the phrase to be corrected.
[0056] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a device for screening error correction phrase candidate items, the device for screening error correction phrase candidate items comprising a memory, a processor, and a screening program for error correction phrase candidate items stored in the memory and executable on the processor, the screening program for error correction phrase candidate items implementing the steps of the method for screening error correction phrase candidate items as described in any one of the above items when executed by the processor.
[0057] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a screening program for error correction phrase candidate items is stored, and when the error correction phrase candidate item screening program is executed by a processor, the steps of the error correction phrase candidate item screening method as described in any of the above items are implemented.
[0058] This embodiment corrects a large number of phrases to be corrected by a secondary candidate correction method, obtains multiple candidates, and uses the primary sorting rules to calculate the primary weight values of all candidates to obtain a system preset number of first candidate candidates, wherein the larger the weight value, the higher the accuracy of the candidate. According to the size of the primary weight value, a part of irrelevant candidates is screened out to prevent all candidates from entering the secondary candidates, and to prevent too many irrelevant options from affecting the accuracy of the secondary sorting model to screen out the correct candidates. Then, the secondary weight value corresponding to the first candidate is obtained and sorted, and the second candidate with the largest weight value is marked as the target candidate to replace the phrase to be corrected to complete the correction, which solves the technical problem of low efficiency of existing error correction technology. The present invention adopts a strategy of primary filtering and secondary scoring, thereby speeding up the system operation efficiency and improving the error correction processing speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the device for screening error correction phrase candidates of the present invention;
[0060] Figure 2A schematic flow chart of a first embodiment of a method for selecting error correction phrase candidates of the present invention;
[0061] Figure 3 A schematic flow chart of a second embodiment of a method for selecting error correction phrase candidates of the present invention;
[0062] Figure 4 for Figure 2 A detailed flow chart of step S20 in an embodiment;
[0063] Figure 5 for Figure 2 A detailed flow chart of step S40 in an embodiment;
[0064] Figure 6 for Figure 2 A detailed flow chart of step S50 in an embodiment;
[0065] Figure 7 It is a functional module diagram of an embodiment of a device for screening error-correcting phrase candidates of the present invention.
[0066] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0067] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0068] The invention provides a device for screening error correction phrase candidate items.
[0069] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the device for screening error correction phrase candidates of the present invention.
[0070] like Figure 1 As shown, the screening device for the error correction phrase candidate items may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0071] Those skilled in the art can understand that Figure 1 the hardware structure of the screening device for error correction phrase candidates shown in does not constitute a limitation on the screening device for error correction phrase candidates, and may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0072] Such as Figure 1 As shown, in the memory 1005 as a computer-readable storage medium, an operating system, a network communication module, a user interface module, and a screening program for error correction phrase candidates may be included. Among them, the operating system is a program for managing and controlling the screening device for error correction phrase candidates and software resources, and supports the operation of the network communication module, the user interface module, the screening program for error correction phrase candidates, and other programs or software; the network communication module is used for managing and controlling the network interface 1004; the user interface module is used for managing and controlling the user interface 1003.
[0073] In Figure 1 In the hardware structure of the screening device for error correction phrase candidates shown, the network interface 1004 is mainly used to connect to the system background and perform data communication with the system background; the user interface 1003 is mainly used to connect to the client (user side) and perform data communication with the client; the screening device for error correction phrase candidates calls the screening program for error correction phrase candidates stored in the memory 1005 through the processor 1001 and executes the operations of the following embodiments of the screening method for error correction phrase candidates.
[0074] Based on the above hardware structure of the screening device for error correction phrase candidates, various embodiments of the screening method for error correction phrase candidates of the present invention are proposed.
[0075] Referring to Figure 2 , Figure 2 is a schematic flowchart of the first embodiment of the screening method for error correction phrase candidates of the present invention. In this embodiment, the screening method for error correction phrase candidates includes the following steps:
[0076] Step S10, read multiple candidates corresponding to the phrase to be error-corrected;
[0077] In this embodiment, the phrase to be error-corrected refers to the misspelled words typed by the user or the incorrect phrases that appear in the text due to language conversion. For example, "water capital" in "Can water be insured?", "fallopian tube" in "Can I buy insurance for fallopian tube surgery?", and again, "lupus erythematosus" in "systemic lupus erythematosus" and other texts to be error-corrected.
[0078] In this embodiment, multiple candidates for the phrase to be corrected refer to the corresponding candidates retrieved from the preset dictionary using the phrase to be corrected as the keyword. For example, taking "水都" in "水都可以投保吗" as the keyword, the corresponding retrieved candidates include "水痘", "水斗", "水豆", etc. Another example is taking "输" as the keyword for retrieval, and the obtained candidates include multiple corresponding candidates such as "输卵管", "输精管", "输血管", "输血", "输了", etc.
[0079] Step S20: Based on the attribute values of each of the candidates, calculate the primary weight value of each of the candidates and sort them to determine the first sorting result of the candidates.
[0080] In this embodiment, calculate the primary weight value of each candidate corresponding to the phrase to be corrected. Further, sort each candidate according to the primary weight value to obtain the corresponding first sorting result. Among them, the sorting method can be arranged from high to low or from low to high according to the weight value, and the specific sorting method is not limited.
[0081] In this embodiment, the larger the primary weight value of each candidate, the greater the possibility that the candidate is the correct candidate, that is, the higher the accuracy of the candidate as the target candidate. According to the size of the weight value, sort all the candidates in the phrase to be corrected, and retain a preset number of candidates with higher accuracy to avoid all candidates entering the secondary candidates, speed up the operation of the system, and avoid too many irrelevant candidates affecting the secondary sorting model to obtain more accurate candidates.
[0082] Step S30: Based on the first sorting result, obtain multiple first alternative candidates corresponding to the phrase to be corrected.
[0083] In this embodiment, according to the obtained first sorting result, obtain the first score of each candidate, sort each candidate according to the score from high to low, and obtain a preset number of candidates. The purpose is to avoid all candidates entering the secondary candidate system to speed up the operation efficiency of the system and avoid the influence of too many irrelevant candidates. For example, a total of 50 candidates are retrieved. Among them, many candidates are irrelevant. Therefore, first score and sort these 50 candidates, and select the candidates ranked in the TOP k (such as the top 10) as the first alternative candidates, and then perform the second round of scoring and sorting to determine the final replacement item to complete the correction of the text to be corrected.
[0084] Furthermore, a preset number of candidates are marked as the first candidate for the word fragment and output. For example, a total of 50 candidates are retrieved, some of which are irrelevant candidates, so the candidates ranked in the TOP k (for example, the top 10) are selected from the 50 candidates as the first candidate, and output for the next step.
[0085] Step S40, calling a preset secondary scoring and sorting model, respectively obtaining the secondary weight values of the first candidate items and sorting them, to obtain a second sorting result of the first candidate items;
[0086] In this embodiment, according to the first candidate item corresponding to the phrase to be corrected, the preset secondary scoring and sorting model is called to obtain the secondary weight value of each first candidate item to obtain the second sorting result. Further, according to the second sorting result, the secondary weight value of each first candidate item is read. If the secondary weight value of the corresponding first candidate item is less than the preset threshold, such as 0.5, the first candidate item is eliminated; if the secondary weight value of the corresponding first candidate item is greater than the preset threshold, the first candidate item is accepted. After this round of elimination, the second candidate item corresponding to the phrase to be corrected is screened out.
[0087] Step S50, based on the second sorting result, obtaining a plurality of second candidate items corresponding to the phrase to be corrected;
[0088] In this embodiment, according to the second sorting result of the first candidate items corresponding to the phrase to be corrected, the secondary weight values corresponding to each first candidate item are obtained, and according to the secondary weight values, each first candidate item is screened to obtain the second candidate item corresponding to the phrase to be corrected. For example, the first candidate items of the phrase "(heating tube)" include "fallopian tube", "vas deferens", "blood tube", "blood transfusion", "lost", "lost money", "input", "transport", etc. These candidates are respectively put into the secondary scoring and sorting model for scoring, and the secondary weight values corresponding to each candidate item are obtained, which are: 0.95, 0.85, 0.7, 0.6, 0.2, 0.2, 0.1, 0.05, respectively. The candidates with secondary weight values less than 0.5 are eliminated, and the four candidates "fallopian tube", "vas deferens", "blood tube", and "blood transfusion" are retained as the second candidate items.
[0089] Step S60: Screen the second candidate items with the highest secondary weight value among the second candidate items, and mark the corresponding second candidate items as target candidates.
[0090] In this embodiment, according to the weight values of the corresponding second alternative candidates, the second alternative candidates with the highest scores are selected and marked as target candidates. For example, the first alternative candidates of the phrase "(heating tube)" include "fallopian tube", "vas deferens", "blood tube", "blood transfusion", "lost", "lost money", "input", "transport", etc. These candidates are respectively put into the secondary scoring and sorting model for scoring, and the secondary weight values corresponding to each candidate are obtained, which are: 0.95, 0.85, 0.7, 0.6, 0.2, 0.2, 0.1, 0.05, respectively. The candidates with secondary weight values less than 0.5 are eliminated, and the four candidates "fallopian tube", "vas deferens", "blood tube", and "blood transfusion" are retained as the second alternative candidates. Further, the four second alternative candidates are sorted in order, and the candidate with the largest weight value, that is, "fallopian tube" with a weight value of 0.95, is marked as the target candidate. Among them, they can be sorted in order from large to small, or in order from small to large.
[0091] This embodiment corrects a large number of phrases to be corrected by a secondary candidate correction method, obtains multiple candidates, calculates the primary weight values of all candidates using the primary sorting rules, and obtains a system-preset number of first candidate candidates, wherein the larger the weight value, the higher the accuracy of the candidate. According to the size of the primary weight value, a part of irrelevant candidates is screened out to prevent all candidates from entering the secondary candidates, and to prevent too many irrelevant options from affecting the accuracy of the secondary sorting model to screen out the correct candidates. Then, the secondary weight value corresponding to the first candidate is obtained and sorted, and the second candidate with the largest weight value is marked as the target candidate to replace the phrase to be corrected to complete the correction, which solves the technical problem of low efficiency of existing error correction technology. The present invention adopts a strategy of primary filtering and secondary scoring, thereby speeding up the system operation efficiency and improving the error correction processing speed.
[0092] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the method for selecting error correction phrase candidates of the present invention is shown in FIG.
[0093] Step S01, obtaining corpus data and using the corpus data as a training sample set;
[0094] In this embodiment, a large amount of original corpus data is collected, and these corpus data contain a lot of relevant industry data or information, such as the disease word "systemic lupus erythematosus" and the like.
[0095] In this embodiment, the original corpus is segmented, and further, the pre-processed data is used as a training sample set. The training sample set is a learning sample data set, which is used to establish a classifier by matching some parameters, that is, the positive and negative samples in the training sample set are used to train the machine learning model to determine the parameters of the machine learning model.
[0096] This embodiment specifically includes preprocessing the original corpus data and using it as a training sample set, so as to effectively train the parameters in the secondary scoring and ranking model, effectively prevent the training results from being biased towards extremes, and make the scoring and ranking effect of the secondary scoring and ranking model obtained by training the original corpus data more accurate.
[0097] Step S02, based on the training sample set, extracting the first parameter feature of the training sample set, wherein the first parameter feature includes a word frequency change feature, a word segmentation change feature and a language model feature;
[0098] In this embodiment, features such as word frequency changes, word segmentation changes, and language models in the training sample set are extracted to further improve the accuracy of the training of the secondary scoring and ranking model.
[0099] Step S03: Based on the first parameter feature of the training sample set, the XGboost algorithm is used to train the training sample set to construct a secondary scoring and ranking model.
[0100] In this embodiment, the training sample set is trained using the XGboost algorithm according to the characteristic parameters of the data in the training sample set to construct a secondary scoring and ranking model.
[0101] In this embodiment, the XGboost (eXtreme Gradient Boosting) algorithm is an improvement on the Boosting algorithm based on the GBDT (GradientBoosting DecisionTree) algorithm. The internal decision tree uses a regression tree. The algorithm output is a set of regression trees, including multiple regression trees. The basic idea of training and learning is to traverse all segmentation methods (that is, node splitting methods) of all features of the training samples, select the segmentation method with the smallest loss, obtain two leaves (that is, split the node to generate a new node), and then continue to traverse until:
[0102] (1) If the stop splitting condition is met, a regression tree is output;
[0103] (2) If the conditions for stopping iteration are met, a set of regression trees is output.
[0104] In this embodiment, the training samples used by the XGboost algorithm are the training sample set composed of the original corpus data, that is, each training sample in this training sample set belongs to the same data source - the preset corpus.
[0105] In the XGboost algorithm, when traversing all splitting methods of all features in the training sample set, the gain value is used to evaluate the pros and cons of the splitting method, and the splitting method with the smallest loss is selected for each split node. Therefore, the gain value of the split node can be used as the evaluation basis for feature importance. The larger the gain value of the split node, the smaller the loss of node splitting, and thus the greater the importance of the feature corresponding to the split node.
[0106] In this embodiment, since there are multiple regression trees in the trained secondary scoring and ranking model, and different regression trees may use the same feature for node splitting, it is necessary to calculate the average gain value of the split nodes corresponding to the same feature in all regression trees included in the secondary scoring and ranking model, and use the average gain value as the score of the corresponding feature.
[0107] Refer to Figure 4 , Figure 4 For Figure 2 a detailed flowchart of an embodiment of step S20 in
[0108] Step S201: Obtain the attribute values of each candidate item respectively, where the attribute values of the candidate item include word frequency, edit distance, and pinyin Jaccard distance;
[0109] In this embodiment, according to multiple candidate items corresponding to the phrase to be corrected, the word frequency, edit distance, and pinyin Jaccard distance of each candidate item are obtained respectively.
[0110] In this embodiment, the edit distance refers to the number of characters at the same position of the original word and the candidate word that are different, and the pinyin Jaccard distance refers to the number of identical letters in the pinyin of word segment 1 and word segment 2 or the number of merged letters in pinyin 1 and 2.
[0111] In this embodiment, the corresponding candidate words include matching entries and / or associated entries. For example, using the phrase "(tube, conduit)" as the keyword, "fallopian tube", "vas deferens", "blood vessel", "blood transfusion", "lost", "lost money", "input", "transport" and other candidate words are retrieved using the inverted index. Among them, "fallopian tube" and "vas deferens" are matching entries, while candidate words such as "blood transfusion", "lost", "lost money", "input", "transport" are associated entries. By counting the number of occurrences of entries in the original corpus, the word frequency corresponding to each retrieved candidate word is obtained.
[0112] Step S202, calculating the primary weight values of the candidate items based on the word frequency, edit distance and pinyin jaccard distance of the candidate items;
[0113] The following formula is used to calculate the primary weight value of the candidate item:
[0114] M=log 10 (T)-PQ;
[0115] Among them, M represents the primary weight value of the candidate, T represents the word frequency of the corresponding candidate, P represents the edit distance of the corresponding candidate, and Q represents the pinyin jaccard distance;
[0116] In this embodiment, the primary weight value of each candidate item can be calculated according to the word frequency information, the edit distance and the pinyin jaccard distance, and the candidate items can be sorted according to the primary weight value.
[0117] In this embodiment, the word frequency of the candidate word is proportional to the weight value, the edit distance of the candidate word is inversely proportional to the weight value, and the pinyin jaccard distance of the candidate word is also inversely proportional to the weight value.
[0118] In this embodiment, the pinyin jaccard distance refers to the number of identical letters between the word fragment pinyin 1 and the word fragment pinyin 2 or the number of letters of pinyin 1 and 2 combined.
[0119] In this embodiment, the word frequency and edit distance of each candidate item are obtained respectively, where the edit distance refers to the number of different characters in the same position between the original word and the candidate word. For example, "Can water beans be reimbursed?" The candidate words for the wrong word "water beans" are "chickenpox", "water city", "water beans" and the candidate item "chickenpox", "water city" have the same number of different characters in the same position, both 1, so their edit distances are both 1.
[0120] Furthermore, since the word frequency of the candidate words is proportional to the weight value, when calculating the primary weight value of each candidate item and sorting them, the word frequency can be directly used as the weight value, or the word frequency can be divided into different intervals according to size, and each interval corresponds to a different weight value. For example, by taking advantage of the relationship that the word frequency information is proportional to the score, the word frequency information is directly used as the weight value, and then the recalled candidate words can be sorted in the order of the word frequency information, and the candidate words with larger word frequency information are placed in front, that is, words with larger word frequency can be displayed first, or they can be sorted in reverse order according to the size of the word frequency. It can be understood that in actual applications, those skilled in the art can flexibly select the sorting method according to needs, and the embodiment of the present invention does not limit the specific method of sorting candidate words according to word frequency information.
[0121] Specifically, the primary weight value T of the candidate can be calculated by the following formula:
[0122] Primary weight value T==log10(word frequency)-edit distance-pinyin jaccard distance
[0123] For example, "Can water beans be reimbursed?" The candidate words for the wrong word "water beans" are "chickenpox" and "water city". Their edit distances are both 1, and the jaccard distances of pinyin are both 0. They can be distinguished based on word frequency; thus, "chickenpox" with a large weight value can be selected.
[0124] Chickenpox 10 (10w)–1–jaccard(dou,dou)=4
[0125] Water City Log 10 (426)–1–jaccard(dou,dou)=1.63
[0126] Furthermore, the introduction of Pinyin Jaccard distance can solve the differences in initials and finals caused by non-standard local pronunciation;
[0127] For example, in the "heating tube surgery reimbursement process", the candidates for the wrong word "heating tube" are "fallopian tube" and "vas deferens"; the word frequencies are both 10w, and the edit distances differ by 1, but their pinyin initial consonants are different, resulting in different pinyin jaccard distances; thus, "fallopian tube" with a large weight value can be given priority.
[0128] Furthermore, since the word frequency of the candidate words is proportional to the weight value, the word frequency information can be directly used as the weight value when calculating the primary weight value of each candidate item. For example, by using the relationship that the word frequency information is proportional to the weight value and directly using the word frequency information as the weight value, the candidate items can be sorted in order of the word frequency information, and the candidate items with larger word frequency information can be placed in front, that is, words with larger word frequency can be displayed first, or they can be sorted in reverse order according to the size of the word frequency. In actual applications, those skilled in the art can flexibly select the sorting method according to their needs, and the embodiment of the present invention does not limit the specific method of sorting the candidate words according to the word frequency information.
[0129] Step S203: based on the primary weight values of the candidate items, sort the candidate items by weight values to obtain a first sorting result.
[0130] In this embodiment, according to the primary weight value of each candidate, the candidate is sorted by weight value to obtain a preset number of candidates, the purpose is to avoid all candidates from entering the secondary candidate system, to speed up the operation efficiency of the system, and to reduce the impact of irrelevant candidates. For example, a total of 50 candidates are retrieved, among which many candidates are irrelevant, so first score and sort these 50 candidates, screen out the candidates ranked in TOP k (such as the top 10), and mark these 10 candidates as the first alternative candidates.
[0131] Further, a preset number of candidates are marked as first candidate options and output. For example, a total of 50 candidates are retrieved, among which some candidates are irrelevant candidates, so the candidates ranked in TOPk (for example, the top 10) are selected from these 50 candidates as first candidate options and output. Further, these first candidate options are sorted according to the weight value to obtain the first sorting result.
[0132] Reference Figure 5 , Figure 5 for Figure 2 Based on the above embodiment, in this embodiment, the above step S40 further includes:
[0133] Step S401, extracting second parameter features of the first candidate items respectively;
[0134] In this embodiment, according to the first candidate items corresponding to the phrase to be corrected, parameter features of the first candidate items are extracted respectively, wherein the parameter features include word frequency change features, word segmentation change features and language model features.
[0135] Step S402: Based on the second parameter feature, call a preset secondary scoring and sorting model to respectively calculate the secondary weight values of the first candidate items;
[0136] In this embodiment, since the word frequency change feature, word segmentation change feature, and language model feature of the candidate items are helpful for error correction, the secondary scoring and ranking model can be called according to these features of the candidate items to calculate the secondary weight values of the first alternative candidate items to be predicted respectively. For example: [Systemic lupus erythematosus insurance], the correct one is [Systemic lupus erythematosus]. If the candidate item is the incorrect [Erythematosus lupus], it can be found that its word frequency is very low, less than 30; while the correct candidate item [Lupus erythematosus] has a very high word frequency of 1688; after performing word segmentation processing on each candidate item and viewing its word segmentation features, it can be found that the word segmentation of the incorrect candidate item [Erythematosus lupus] is very scattered, which is 1\1\2, that is, [Red\Erythematosus\Lupus]; the correct word segmentation granularity is very large, which is (4), that is, the four characters [Lupus erythematosus] form a word; further, when using the language model to predict the character "spot", it can be found that the incorrect "erythematosus" is not in the top 20, while the correct "spot" ranks first and the probability is close to 1, which is 0.94.
[0137] In this embodiment, each first alternative candidate item is put into the secondary scoring and ranking model. For example, the first alternative candidate items of the phrase "(Fallopian tube)" include "Fallopian tube", "Vas deferens", "Blood vessel", "Blood transfusion", "Blood transfusion", "Lost", "Lost money", "Input", "Transport", etc. These candidate items are respectively put into the secondary scoring and ranking model for prediction to obtain the secondary weight values corresponding to each candidate item, which are: 0.95, 0.6, 0.8, 0.65, 0.3, 0.3, 0.1, 0.05.
[0138] Step S403, sort the first alternative candidate items according to the preset order according to the secondary weight values of the first alternative candidate items to obtain the second sorting result of the first alternative candidate items.
[0139] In this embodiment, all the first alternative candidate items are sorted according to the preset order according to the secondary weight values of the first alternative candidate items to obtain the corresponding second sorting result. For example, the first alternative candidate items of the phrase "(Fallopian tube)" include "Fallopian tube", "Vas deferens", "Blood vessel", "Blood transfusion", "Blood transfusion", "Lost", "Lost money", "Input", "Transport", etc. The secondary weight values of each candidate item are: 0.95, 0.6, 0.8, 0.65, 0.3, 0.3, 0.1, 0.05. Sort them to obtain the corresponding second sorting result.
[0140] In this embodiment, the preset order can be from large to small, or from small to large. For example, "fallopian tube" (0.95) > "blood transfusion tube" (0.8) > "blood transfusion" (0.65) > "vas deferens" (0.6) > "lost" (0.3) = "lost money" (0.3) > "input" (0.1) > "transport" (0.05), or "transport" (0.05) < "input (0.1)" < "lost money" (0.3) = "lost" (0.3) < "vas deferens" (0.6) < "blood transfusion" (0.65) < "vas deferens" (0.8) < "fallopian tube" (0.95).
[0141] Reference Figure 6 , Figure 6 for Figure 2 Based on the above embodiment, in this embodiment, the above step S50 further includes:
[0142] Step S501, screening the first candidate items based on the second sorting result;
[0143] In this embodiment, according to the second sorting result, the secondary weight values of all the first candidate items are respectively obtained, and according to the secondary weight values of the respective first candidate items, all the first candidate items are screened.
[0144] For example, the first candidate of the phrase "(heating tube)" includes "fallopian tube", "vas deferens", "blood transfusion tube", "blood transfusion", "lost", "lost money", "input", "transport", etc. The corresponding second ranking results are "fallopian tube" (0.95) > "blood transfusion tube" (0.8) > "blood transfusion" (0.65) > "vas deferens" (0.6) > "lost" (0.3) = "lost money" (0.3) > "input" (0.1) > "transport" (0.05), or "transport" (0.05) < "input (0.1)" < "lost money" (0.3) = "lost" (0.3) < "vas deferens" (0.6) < "blood transfusion" (0.65) < "blood transfusion tube" (0.8) < "fallopian tube" (0.95). Among them, the scores of each first candidate are 0.95, 0.6, 0.8, 0.65, 0.3, 0.3, 0.1, 0.05 respectively.
[0145] In this embodiment, according to the secondary weight values corresponding to the first alternative candidates, the corresponding first alternative candidates are eliminated, and then multiple second alternative candidates corresponding to the phrase to be corrected are screened out from them. For example, the first alternative candidates for the phrase "(Fallopian tube)" include "Fallopian tube", "vas deferens", "blood vessel for transfusion", "blood transfusion", "lost", "lost money", "input", "transport", etc. The secondary weight values of each alternative candidate are: 0.95, 0.6, 0.8, 0.65, 0.3, 0.3, 0.1, 0.05. According to the secondary weight values of these first alternative candidates, these candidates are screened.
[0146] Step S502, if there is a secondary weight value of a first alternative candidate less than the preset threshold, then eliminate the first alternative candidate;
[0147] In this embodiment, according to the secondary weight values of each first alternative candidate, each first alternative candidate is screened. If the secondary weight value corresponding to the first alternative candidate is less than the preset value, then the corresponding first alternative candidate is eliminated. For example, taking the pinyin (dajiang) as the keyword, the corresponding first alternative candidates are "big prize", "general", "DJI", "big river", "big sauce". Calculate the weight values of each first alternative candidate respectively. Among them, the weight values of "big prize", "general", and "big sauce" are 0.1, 0.15, and 0.3 respectively, all less than 0.5, so these three first alternative candidates are eliminated from them.
[0148] Step S503, if there is a secondary weight value of a first alternative candidate greater than the preset threshold, then retain the first alternative candidate;
[0149] In this embodiment, if the corresponding secondary weight value is greater than the preset threshold, then retain the first alternative candidate, and at the same time mark the corresponding first alternative candidate as a second alternative candidate. For example, taking the pinyin (da jiang) as the keyword, the recalled candidate words are "big prize", "general", "DJI", "big river", "big sauce". Using the scoring and ranking model, calculate the weight values of each candidate word respectively. Among them, the weight values of "DJI" and "big river" are 0.85 and 0.6 respectively, assuming both are greater than the preset value of 0.5, then accept the corresponding first alternative candidate. For example, for "moderate anemia cephalosporin", the pinyin "toubao" of "cephalosporin" can be used as the keyword to recall multiple corresponding candidates, and eliminate the first alternative candidates less than the preset value of 0.5. The secondary weight values of the two first alternative candidates "insurance", "cephalosporin" are 0.95 and 0.7 respectively, which are greater than the preset value of 0.5, so accept these two first alternative candidates.
[0150] Step S504, based on the first alternative candidate, mark the corresponding first alternative candidate as a second alternative candidate.
[0151] In this embodiment, multiple first alternative candidate items whose secondary weight values are greater than the preset value are marked as second alternative candidate items. For example, for "moderate anemia cephalosporin", the pinyin "tou bao" of "cephalosporin" can be used as a keyword to recall multiple corresponding candidate items, and the first alternative candidate items whose secondary weight values are less than the preset value of 0.5 are eliminated. The secondary weight values of the two first alternative candidate items "insurance, cephalosporin" are 0.95 and 0.7 respectively, which are greater than the preset value of 0.5, and the two first alternative candidate items are accepted. Further, the two first alternative candidate items "insurance, cephalosporin" are marked as second alternative candidate items.
[0152] Furthermore, after the above step S60, the following steps are further included:
[0153] Step S70, replacing the phrase to be corrected with the target candidate as a replacement item;
[0154] In this embodiment, the target candidate is marked as a replacement item. For example, in the question "Can I get insurance for a heating tube surgery?", the target candidate of the phrase "heating tube" to be corrected is "fallopian tube", and "fallopian tube" is marked as a replacement item.
[0155] In this embodiment, according to the replacement item, the replacement item is replaced with the corresponding phrase to be corrected, and the correction of the text to be corrected is completed. For example, after two screenings, the multiple candidate items corresponding to the phrase to be corrected "heating tube" in "Can I buy insurance for heating tube surgery?", the final target candidate item is "fallopian tube", and this target candidate item "heating tube" is marked as a replacement item, and further replaced with the phrase to be corrected "heating tube" in "Can I buy insurance for heating tube surgery?" to obtain "Can I buy insurance for fallopian tube surgery?", completing the correction of the phrase to be corrected.
[0156] Reference Figure 7 , Figure 7 This is a functional module diagram of an embodiment of a device for selecting error correction phrase candidates of the present invention. In this embodiment, the device for selecting error correction phrase candidates includes:
[0157] A reading module 10, used for reading a plurality of candidate items corresponding to the phrase to be corrected;
[0158] A calculation module 20, configured to calculate a primary weight value of each candidate item based on the attribute value of each candidate item and sort the items to determine a first sorting result of the candidate items;
[0159] A first acquisition module 30, configured to acquire a first candidate item corresponding to the phrase to be corrected based on the first sorting result;
[0160] The secondary calculation module 40 is used to call a preset secondary scoring and ranking model to respectively calculate the secondary weight values of the first candidate items and sort them to obtain a second ranking result of the first candidate items;
[0161] A second acquisition module 50, configured to acquire a plurality of second candidate items corresponding to the phrase to be corrected based on the second sorting result;
[0162] The marking module 60 is used to select the second candidate item with the highest secondary weight value among the second candidate items, and mark the corresponding second candidate item as a target candidate item.
[0163] Optionally, in a specific embodiment, the device for screening error correction phrase candidates further comprises:
[0164] A third acquisition module is used to acquire corpus data and use the corpus data as a training sample set;
[0165] An extraction module, configured to extract a first parameter feature of the training sample set based on the training sample set;
[0166] A construction module is used to train the training sample set using an XGboost algorithm based on the first parameter feature of the training sample set to construct a secondary scoring and ranking model.
[0167] Optionally, in a specific embodiment, the calculation module is specifically used to:
[0168] Acquire the attribute value of each candidate item respectively, wherein the attribute value of the candidate item includes word frequency, edit distance and pinyin jaccard distance; calculate the primary weight value of the candidate item respectively based on the word frequency, edit distance and pinyin jaccard distance of the candidate item; sort the candidate items according to the weight value based on the primary weight value of the candidate item to obtain a first sorting result.
[0169] Optionally, in a specific embodiment, the secondary calculation module is specifically used for:
[0170] Extract second parameter features of the first alternative candidate items respectively, wherein the second parameter features include word frequency change features, word segmentation change features and language model features; based on the second parameter features, call a preset secondary scoring and sorting model to measure the first alternative candidate items respectively to determine the secondary weight value of the first alternative candidate items; according to the secondary weight value of the first alternative candidate items, sort the first alternative candidate items in a preset order to obtain a second sorting result of the first alternative candidate items.
[0171] Optionally, in a specific embodiment, the second acquisition module is specifically used to:
[0172] Based on the second sorting result, the first alternative candidate items are screened; when the secondary weight value of the first alternative candidate item is less than the preset threshold, the first alternative candidate item is eliminated; when the secondary weight value of the first alternative candidate item is greater than the preset threshold, the first alternative candidate item is retained; and the first alternative candidate item retained after screening is marked as the second alternative candidate item.
[0173] Furthermore, the device for screening error correction phrase candidates further comprises:
[0174] A replacement module is used to replace the to-be-corrected phrase with the target candidate item as a replacement item.
[0175] This embodiment corrects a large number of phrases to be corrected by a secondary candidate correction method, obtains multiple candidates, calculates the primary weight values of all candidates using the primary sorting rules, and obtains a system-preset number of first candidate candidates, wherein the larger the weight value, the higher the accuracy of the candidate. According to the size of the primary weight value, a part of irrelevant candidates is screened out to prevent all candidates from entering the secondary candidates, and to prevent too many irrelevant options from affecting the accuracy of the secondary sorting model to screen out the correct candidates. Then, the secondary weight value corresponding to the first candidate is obtained and sorted, and the second candidate with the largest weight value is marked as the target candidate to replace the phrase to be corrected to complete the correction, which solves the technical problem of low efficiency of existing error correction technology. The present invention adopts a strategy of primary filtering and secondary scoring, thereby speeding up the system operation efficiency and improving the error correction processing speed.
[0176] Based on the same embodiment description as the above-mentioned method for selecting error-correcting phrase candidate items of the present invention, the embodiment description of the device for selecting error-correcting phrase candidate items will not be elaborated in detail in this embodiment.
[0177] The present invention also provides a computer-readable storage medium.
[0178] In this embodiment, the computer-readable storage medium stores a screening program for error correction phrase candidate items, and when the screening program for error correction phrase candidate items is executed by the processor, the steps of the screening method for error correction phrase candidate items described in any of the above embodiments are implemented. The method implemented when the screening program for error correction phrase candidate items is executed by the processor can refer to the various embodiments of the screening method for error correction phrase candidate items of the present invention, so it will not be described in detail.
[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0180] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
Claims
1. A method for screening error correction phrase candidates, characterized in that: The method for screening error correction phrase candidate items comprises the following steps: Read multiple candidate items corresponding to the phrase to be corrected; Respectively obtain the attribute value of each candidate item, wherein the attribute value of the candidate item includes word frequency, edit distance and pinyin jaccard distance; Based on the word frequency, edit distance and pinyin jaccard distance of the candidate items, respectively calculate the primary weight values of the candidate items; The following formula is used to calculate the primary weight value of the candidate item: M= log 10 (T)-P-Q; Among them, M represents the primary weight value of the candidate, T represents the word frequency of the corresponding candidate, P represents the edit distance of the corresponding candidate, and Q represents the pinyin jaccard distance; Based on the primary weight values of the candidate items, the candidate items are sorted according to the weight values to obtain a first sorting result; Based on the first sorting result, obtaining a plurality of first candidate items corresponding to the phrase to be corrected; Calling a preset secondary scoring and sorting model to respectively measure and sort the secondary weight values of the first candidate items, thereby obtaining a second sorting result of the first candidate items; Based on the second sorting result, obtaining a plurality of second candidate items corresponding to the phrase to be corrected; The second alternative candidate items with the highest secondary weight value among the second alternative candidate items are screened, and the corresponding second alternative candidate items are marked as target candidate items.
2. The method for selecting error correction phrase candidates according to claim 1, wherein: Before the step of reading a plurality of candidate items corresponding to the phrase to be corrected, the method further includes: Acquire corpus data and use the corpus data as a training sample set; Based on the training sample set, extracting first parameter features of the training sample set, wherein the first parameter features include word frequency change features, word segmentation change features and language model features; Based on the first parameter feature of the training sample set, the training sample set is trained using the XGboost algorithm to construct a secondary scoring and ranking model.
3. The method for selecting error correction phrase candidates according to claim 1, wherein: The calling of the preset secondary scoring and sorting model to respectively calculate the secondary weight values of the first candidate items and sort them to obtain the second sorting result of the first candidate items includes: Extracting second parameter features of the first candidate items respectively, wherein the second parameter features include word frequency change features, word segmentation change features and language model features; Based on the second parameter feature, calling a preset secondary scoring and ranking model to respectively calculate the secondary weight values of the first candidate items; According to the secondary weight values of the first alternative candidate items, the first alternative candidate items are sorted in a preset order to obtain a second sorting result of the first alternative candidate items.
4. The method for selecting error correction phrase candidates according to claim 1, wherein: The obtaining, based on the second sorting result, a plurality of second candidate items corresponding to the phrase to be corrected comprises: Based on the second sorting result, screening each of the first candidate items; If there is a first candidate item whose secondary weight value is less than the preset threshold, the first candidate item is eliminated; If there is a first alternative candidate item whose secondary weight value is greater than the preset threshold, the first alternative candidate item is retained; The first alternative candidate items retained after screening are marked as second alternative candidates.
5. The method for selecting error correction phrase candidates according to claim 1, wherein: After the step of screening the second candidate items with the highest secondary weight value among the second candidate items and marking the corresponding second candidate items as target candidate items, the method further includes: The target candidate item is used as a replacement item to replace the phrase to be corrected.
6. A device for selecting candidate error correction phrases, characterized in that: The device for screening error correction phrase candidate items comprises: A reading module, used for reading multiple candidate items corresponding to the phrase to be corrected; The calculation module is used to obtain the attribute value of each candidate item respectively, wherein the attribute value of the candidate item includes word frequency, edit distance and pinyin jaccard distance; based on the word frequency, edit distance and pinyin jaccard distance of the candidate item, the primary weight value of the candidate item is calculated respectively; wherein the primary weight value of the candidate item is calculated using the following formula: M=log 10 (T)-PQ; where M represents the primary weight value of the candidate item, T represents the word frequency of the corresponding candidate item, P represents the edit distance of the corresponding candidate item, and Q represents the pinyin jaccard distance; based on the primary weight value of the candidate item, the candidate items are sorted according to the weight value to obtain a first sorting result; A first acquisition module, configured to acquire a first candidate item corresponding to the phrase to be corrected based on the first sorting result; A secondary calculation module, used to call a preset secondary scoring and ranking model, respectively calculate the secondary weight values of the first candidate items and sort them, and obtain a second ranking result of the first candidate items; A second acquisition module, configured to acquire a plurality of second candidate items corresponding to the phrase to be corrected based on the second sorting result; The marking module is used to select the second alternative candidate items with the highest secondary weight value among the second alternative candidate items, and mark the corresponding second alternative candidate items as target candidate items.
7. The device for selecting error correction phrase candidates according to claim 6, wherein: The device for screening error correction phrase candidate items further comprises: A corpus acquisition module, used to acquire corpus data and use the corpus data as a training sample set; An extraction module, configured to extract a first parameter feature of the training sample set based on the training sample set; A construction module is used to train the training sample set using an XGboost algorithm based on the first parameter feature of the training sample set to construct a secondary scoring and ranking model.
8. A device for screening candidate error correction phrases, characterized in that: The device for screening error correction phrase candidate items includes a memory, a processor, and a screening program for error correction phrase candidate items stored in the memory and executable on the processor. When the screening program for error correction phrase candidate items is executed by the processor, the steps of the method for screening error correction phrase candidate items as described in any one of claims 1-5 are implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a screening program for error-correcting phrase candidate items, and when the screening program for error-correcting phrase candidate items is executed by a processor, the steps of the method for screening error-correcting phrase candidate items according to any one of claims 1 to 5 are implemented.
