Text comparison method, device, equipment and medium
Through the error correction technology of optical character recognition and preset language model group, the editing distance is calculated to judge the changes in text content, which solves the problems of low text verification efficiency and risk of missed search in the existing technology, and achieves high accuracy and high Text verification of efficiency.
Patent Information
- Application Number
- CN202210283527.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-03-22
AI Technical Summary
In the prior art, the verification text is inefficient and there is a risk of missed checking, especially in texts with a large number of pages or terms, it is difficult to accurately verify whether it is changed.
Convert image text into recognition text through optical character recognition, and use preset language model groups (such as bert4csc and macbert) to correct errors, calculate editing distances to judge changes in text content, and possible changes in positioning marks or recognition errors.
It improves the accuracy of text verification, reduces the investment in manpower and material resources, reduces the risk of missed inspections, and improves the verification efficiency of large texts.
Smart Images

Figure CN114611495B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text comparison method, device, equipment and medium. Background Art
[0002] Since the Internet has the advantages of fast data transmission speed and high information exchange efficiency, more and more companies or individuals use the Internet to transmit text. For example, Company A will send the written electronic or PDF version of the text to Company B via the Internet. Company B will print out the received text, sign and seal it, and Company B will send the signed and sealed text to Company A by scanning, uploading or mailing.
[0003] After receiving the signed text returned by the other party, Company A needs to spend a lot of manpower and material resources to compare the signed text with the original document content to verify whether the text content has been modified. Especially for some texts with a large number of pages and clauses, it is difficult to achieve accurate verification, which is not only inefficient, but also has the risk of missing detection. Summary of the invention
[0004] In view of the above, the present invention provides a text comparison method, device, equipment and medium, which aims to solve the technical problems in the prior art of low efficiency in text verification and the risk of missed detection.
[0005] To achieve the above object, the present invention provides a text comparison method, which comprises:
[0006] Receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain error-corrected text;
[0007] Calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain first edit times of character strings at different positions in the proofread text and the recognized text;
[0008] When the first editing times of the character strings at different positions are greater than a preset value, calculating the second editing times of the character strings at different positions of the proofreading text and the error correction text according to a second preset editing distance formula;
[0009] Based on the relationship between the second number of edits and the preset value, it is determined that the character strings at different positions in the recognized text are modified content or incorrectly recognized content, and the character strings at different positions are positioned and marked in the proofreading text and the recognized text, and the proofreading text and the recognized text with the positioning marks are fed back to the user.
[0010] Preferably, the preset language model group includes a first pre-trained language model and a second pre-trained language model, the first pre-trained language model is a BERT4CSC pre-trained language model, the second pre-trained language model is a MacBERT pre-trained language model, and the recognition text is input into the preset language model group for error correction to obtain the error correction text, including:
[0011] Input the recognition text into the first pre-trained language model and the second pre-trained language model respectively for word segmentation to obtain a plurality of words, and calculate each word according to a preset error correction text formula of the preset language model group to obtain a target probability of all predicted words corresponding to each word;
[0012] A predicted word with the maximum target probability is selected from all predicted words corresponding to each word, and each word is replaced according to the predicted word with the maximum target probability to obtain the error correction text.
[0013] Preferably, the preset error correction text formula includes:
[0014] +
[0015] in, is the error correction text, for the recognized text in the first pre-trained language model, For the recognized text in the second pre-trained language model, , is the weighting coefficient, and softmax is the normalization function.
[0016] Preferably, before calculating the proofreading text and the recognized text according to the first preset edit distance formula, the method further includes:
[0017] All special characters of the proofreading text, the recognition text and the error correction text are standardized in a unified format according to a pre-constructed special character dictionary.
[0018] Preferably, the calculating the second edit times of the character strings of the proofreading text and the error correction text at the different positions according to the second preset edit distance formula comprises:
[0019] Read the character strings of the proofreading text and the error correction text at different positions to compare their lengths;
[0020] Calculating the length of the character string of the error correction text according to the second preset edit distance formula and transforming it into the length of the character string of the proofreading text to obtain the number of edits for different operation modes;
[0021] The minimum number of edits among the different operation modes is selected as the second number of edits of the character strings of the proofreading text and the error correction text at the different positions.
[0022] Preferably, the second preset edit distance formula includes:
[0023]
[0024] in, The proofread text The first character of the recognition text and the first character of the error correction text The number of edits of characters, The proofread text characters, The first characters, The first characters, This is the second conditional statement.
[0025] Preferably, the positioning and marking the character strings at different positions in the proofreading text and the recognition text includes:
[0026] When the second editing number is less than or equal to the preset value, the character strings at different positions in the proofreading text and the recognition text are first marked according to a preset marking method;
[0027] When the second editing times is greater than the preset value, the character strings at different positions are second marked in the proofreading text and the recognition text according to a preset marking method.
[0028] To achieve the above object, the present invention further provides a text comparison device, the device comprising:
[0029] Error correction module: used for receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain corrected text;
[0030] A calculation module: used for calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain a first edit number of character strings at different positions in the proofread text and the recognized text;
[0031] Output module: used for calculating the second editing times of the strings of the proofreading text and the error correction text at the different positions according to a second preset editing distance formula when the first editing times of the strings at the different positions are greater than a preset value;
[0032] Feedback module: used to determine whether the character strings at different positions in the recognized text are modified content or incorrectly recognized content based on the size relationship between the second number of edits and the preset value, mark the character strings at different positions in the proofreading text and the recognized text, and feed back the marked proofreading text and the recognized text to the user.
[0033] To achieve the above object, the present invention further provides an electronic device, the electronic device comprising:
[0034] at least one processor; and,
[0035] a memory communicatively connected to the at least one processor; wherein,
[0036] The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor so that the at least one processor can execute the above-mentioned text comparison method.
[0037] To achieve the above object, the present invention also provides a computer-readable medium, wherein the computer-readable medium stores a text comparison program, and when the text comparison program is executed by a processor, the steps of the above text comparison method are implemented.
[0038] The present invention converts the scanned text signed by the other party into a recognized text through optical character recognition, and then inputs the recognized text into a preset language model group for error correction to obtain a relatively high accuracy error correction text. When the first preset edit distance formula calculates that the first edit distance of the character strings at different positions in the proofreading text and the recognized text is greater than the preset value, the error correction text is used as an auxiliary judgment text to calculate the second edit distance of the character strings at the different positions of the proofreading text and the error correction text, and according to the relationship that the second edit distance is less than or equal to the preset value, the character string of the recognized text is determined to be the content of the recognition error. According to the relationship that the second edit distance is greater than the preset value, the character string of the recognized text is determined to be the content that has been modified. According to the relationship between the second number of edits and the preset value, the character strings at different positions in the proofreading text and the recognized text are positioned and marked and fed back to the user.
[0039] The present invention inputs the recognition text into a preset language model group to obtain a correction text with a relatively high accuracy rate, uses the correction text as an auxiliary judgment text, and utilizes the correction text to judge during the verification process of the recognition text by proofreading the text. According to the relationship between the second editing times and the preset value, it is judged whether there is a recognition error or a problem that the character string of the recognition text has been modified, thereby improving the accuracy of the verified text. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A flowchart diagram of a preferred embodiment of the text comparison method of the present invention;
[0041] Figure 2 A schematic diagram of a module of a preferred embodiment of the text comparison device of the present invention;
[0042] Figure 3 A schematic diagram of a preferred embodiment of the electronic device of the present invention;
[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0045] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0046] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0047] The present invention provides a text comparison method. Figure 1 FIG. 2 is a flow chart of a method of an embodiment of a text comparison method of the present invention. The method can be executed by an electronic device, and the electronic device can be implemented by software and / or hardware. The text comparison method includes:
[0048] Step S10: receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain corrected text.
[0049] In this embodiment, the image text refers to the text signed by the other party (e.g., Enterprise B) received by the sending end (e.g., Enterprise A) (e.g., a paper contract or scanned text after signing or stamping). The proofreading text refers to the backup text of the text to be signed by the sending end (e.g., Enterprise A) to the other party (e.g., Enterprise B) (e.g., when Enterprise A sends Text A to Enterprise B, Enterprise A retains a copy of Text A as a backup file), which is stored in an electronic document format and is also called an original file. After receiving the image text signed by Enterprise B, the user of Enterprise A inputs the image text and the proofreading text into the verification system, and the optical character recognition module (OCR module) of the verification system recognizes the text of the image text to obtain the recognized text. The recognized text refers to converting the text of the photocopied document into a text format by optical recognition. Since the optical character recognition technology has a certain error in the recognition of text, it is easy to cause typos in the recognized text. Therefore, the recognized text is input into the preset language model group, and the wrongly recognized words in the recognized text are detected according to the preset error correction text formula of the preset language model group, and the wrongly recognized words are replaced according to the predicted words of the vocabulary of the preset language model group to obtain a relatively high accuracy error correction text.
[0050] In one embodiment, the preset language model group includes a first pre-trained language model and a second pre-trained language model, the first pre-trained language model is a BERT4CSC pre-trained language model, the second pre-trained language model is a MacBERT pre-trained language model, and the recognition text is input into the preset language model group for error correction to obtain the error correction text, including:
[0051] Input the recognition text into the first pre-trained language model and the second pre-trained language model respectively for word segmentation to obtain a plurality of words, and calculate each word according to a preset error correction text formula of the preset language model group to obtain a target probability of all predicted words corresponding to each word;
[0052] A predicted word with the maximum target probability is selected from all predicted words corresponding to each word, and each word is replaced according to the predicted word with the maximum target probability to obtain the error correction text.
[0053] In one embodiment, the preset error correction text formula includes:
[0054] +
[0055] in, is the error correction text, for the recognized text in the first pre-trained language model, For the recognized text in the second pre-trained language model, , is the weighting coefficient, and softmax is the normalization function.
[0056] The preset Chinese word segmentation tools of the pre-language model group include, but are not limited to, word segmentation tools such as LTP or BPE. The preset language model group includes a first pre-trained language model and a second pre-trained language model. The first pre-trained language model and the second pre-trained language model are the pre-trained language models of bert4csc and macbert respectively. The bert4csc pre-trained language model is a model that calculates the similarity of the forward and backward bidirectional strings of the text sequence of the input text through a masking strategy to predict the probability of the next word. The macbert pre-trained language model is a model that calculates the probability of the next word based on the semantic degree of the input text (such as synonyms, antonyms) through a masking strategy. Combining the two pre-trained language models effectively solves the misrecognition of text at the initial stage, which is also an improvement of the present invention on the existing OCR text recognition, and improves the error correction rate of the recognized text.
[0057] The purpose of adding a masking strategy in the training stage of the model is to predict the probability of which predicted words will appear for the input word of the input model. When inputting, the input word needs to be replaced with [mask]. In the inference stage of the model, the text sequence containing typos is input into the model, and the output is the probability of multiple predicted words corresponding to each word in the predicted text sequence. If the probability of the predicted word with the highest probability in the output is greater than the probability of the word, then the word is determined to be a typo.
[0058] For example, the text sequence of the recognized text is "Select a date to enter the site for construction". This text sequence is respectively input into the pre-trained language models of bert4csc and macbert, and all the predicted words corresponding to each word in the text sequence are obtained (for example, when the input word is "site", all the predicted words corresponding to "site" are "factory", "site", "length", "mine", etc.). According to the preset error correction text formula, the normalization function is applied to all the predicted words corresponding to each word in the text sequence to obtain the initial probability of all the predicted words (for example, in the pre-trained language model of bert4csc, the multiple predicted words corresponding to "site" and the initial probability of the predicted words are: "factory" is 73.01%, "site" is 21.21%, "length" is 2.01%, "mine" is 0.51%, etc., and the sum of the initial probabilities of all the predicted words corresponding to "site" is 1).
[0059] The initial probabilities of all predicted words corresponding to each word are obtained from the pre-trained language models of bert4csc and macbert, and the target probabilities of all predicted words corresponding to each word are obtained by weighted averaging. For example, after weighted averaging the initial probabilities of multiple predicted words corresponding to "场", the target probabilities of all predicted words are as follows: "厂" is 59.82%, "场" is 37.12%, "长" is 1.01%, "矿" is 0.31%, etc. Since the target probability of "厂" is greater than the target probability of the input word "场", it indicates that "场" is a misspelled word. Therefore, the predicted word "厂" with the maximum target probability is selected from all predicted words to replace "场" in the text sequence. If the predicted word with the maximum target probability is the word in the input text sequence, it indicates that the word in the input text sequence is not a misspelled word. According to the above method, each word in the text sequence is compared with the target probabilities of all predicted words corresponding to that word to obtain the corrected text.
[0060] Step S20: Calculate the proofreading text and the recognition text according to the first preset edit distance formula to obtain the first edit times of the strings at different positions in the proofreading text and the recognition text.
[0061] Specifically, step S20 includes:
[0062] Read the strings at different positions in the proofreading text and the recognition text for length comparison;
[0063] According to the first preset edit distance formula, calculate the length transformation of the string of the recognition text into the string of the proofreading text to obtain the edit times of different operation methods;
[0064] Select the minimum edit times among the different operation methods as the first edit times of the strings at different positions in the proofreading text and the recognition text.
[0065] In one embodiment, the first preset edit distance formula includes:
[0066]
[0067] Where is the edit times of the th character of the proofreading text and the th character of the recognition text and the corrected text, is the th character of the proofreading text, is the th character of the recognition text, is the first conditional statement.
[0068] The first edit distance matrix of the proofread text and the recognized text is created according to the first preset edit distance formula. If the string of the proofread text is M and the string of the recognized text is N, the dimension of the first edit distance matrix is (M+1)*(N+1). First, assign values to the first row and first column of the matrix sequence, starting from 0, and calculate the first column, the second column, and so on. Compare the characters in all positions and calculate the entire matrix. The calculation rule is: ,in, Indicates an increase operation, Indicates a delete operation. It is represented as a replacement operation, and the number of edits of the three operation modes is calculated. The minimum number of edits among the three operation modes is selected as the first number of edits of the string at all positions in the proofread text and the recognized text. The number of edits refers to the minimum number of edit operations required to convert one character into another between two characters. The larger their distance or the number of operations, the more different the two characters are. On the contrary, the smaller their distance or the number of operations, the more similar the two characters are.
[0069] In one embodiment, before calculating the proofreading text and the recognized text according to the first preset edit distance formula, the method further includes:
[0070] All special characters of the proofreading text, the recognition text and the error correction text are standardized in a unified format according to a pre-constructed special character dictionary.
[0071] The special character dictionary is a dictionary for correcting special characters that may be wrong in the text. All special characters (such as ¥, ;, !, etc.) of the proofreading text, recognition text, and error correction text are standardized, so that the special characters of these three texts are in a unified format, which can improve the calculation efficiency of the edit distance between texts.
[0072] For example, the special character at the same position in the recognized text and the error correction text is "¥", and the special character at the same position in the proofreading text is "¥". According to the special character dictionary, the special character "¥" at the same position in the recognized text and the error correction text is uniformly converted to "¥". After the special characters are unified in format, the number of operations between text distances can be reduced, which is conducive to calculating the editing distance of characters other than special characters.
[0073] Step S30: When the first editing times of the character strings at different positions are greater than a preset value, the second editing times of the character strings at different positions of the proofreading text and the error correction text are calculated according to a second preset editing distance formula.
[0074] In this embodiment, when the first editing times of the character strings at different positions in the proofreading text and the recognized text are greater than a preset value (for example, the preset value is 1), it means that the similarity difference between the character strings at different positions is relatively large, and multiple editing operations are required to obtain the same character string length (similarity). It also means that the character strings at the recognized text at different positions may have recognition errors or have been modified, so these contents have not passed the verification.
[0075] In the case where the verification fails, the proofreading text and the error correction text at all positions are read for length comparison, and the second editing times of the strings of the proofreading text and the error correction text are calculated according to the second preset editing distance formula. When the first editing times are greater than the preset value, the error correction text is introduced as an auxiliary judgment text to judge whether there is a recognition error in the string of the recognized text. If the second editing times are greater than the preset value, it means that there is no recognition error in the string of the recognized text, but the string of the recognized text may have been modified.
[0076] In addition, when the first editing times of the character strings at different positions in the proofreading text and the recognized text are less than a preset value (for example, the preset value is 1), it means that the similarity of the character strings at different positions is very close, which also means that the contents of the recognized text and the proofreading text at the different positions are the same, there is no recognition error or modification, and therefore these contents have passed the verification.
[0077] In one embodiment, the calculating the second edit times of the character strings of the proofreading text and the error correction text at the different positions according to the second preset edit distance formula includes:
[0078] Read the character strings of the proofreading text and the error correction text at different positions to compare their lengths;
[0079] Calculating the length of the character string of the error correction text according to the second preset edit distance formula and transforming it into the length of the character string of the proofreading text to obtain the number of edits for different operation modes;
[0080] The minimum number of edits among the different operation modes is selected as the second number of edits of the character strings of the proofreading text and the error correction text at the different positions.
[0081] In one embodiment, the second preset edit distance formula includes:
[0082]
[0083] in, The proofread text The first character of the recognition text and the first character of the error correction text The number of edits of characters, The proofread text characters, The first characters, The first characters, This is the second conditional statement.
[0084] In this implementation, in the process of calculating the second editing number, if If the content is the same as the proofread text, it means that the content of the recognized text is the same as that of the proofread text, and there is no recognition error or modification. Therefore, these contents have passed the verification and there is no need to consider using .like If they are different, it means that a word somewhere in the recognized text is different or inconsistent and is checked out, but we cannot be sure whether it is caused by an error in optical character recognition or the word has been modified. Therefore, the error correction text is introduced as an auxiliary judgment text to determine whether there is a recognition error or modification in the character string of the recognized text.
[0085] In other implementations, the present invention does not require the first preset edit distance formula to calculate the first edit count, but directly calculates the edit counts between the proofreading text and the recognition text and the correction text respectively based on the second conditional statement of the second preset edit distance formula, thereby improving the system operation time and obtaining the edit counts between different texts more quickly.
[0086] The second edit distance matrix of the proofreading text and the error correction text is created according to the second preset edit distance formula. If the string of the proofreading text is M and the string of the error correction text is K, the dimension of the second edit distance matrix is (M+1)*(K+1). First, assign values to the first row and first column of the matrix sequence, starting from 0, and calculate the first column, the second column, and so on. Compare the characters in all positions and calculate the entire matrix. The calculation rule is: ,in, Indicates an increase operation, Indicates a delete operation. It is represented as a replacement operation, and the editing times of the three operation modes are calculated. The minimum editing times among the editing times of the three operation modes is selected as the second editing times of the character strings at all positions in the proofreading text and the error correction text.
[0087] Step S40: Based on the relationship between the second number of edits and the preset value, determine whether the character strings at different positions in the recognized text are modified content or incorrectly recognized content, mark the character strings at different positions in the proofreading text and the recognized text, and feed back the marked proofreading text and the recognized text to the user.
[0088] In this embodiment, when the second number of edits is less than or equal to a preset value, it indicates that the similarity difference between the character strings at different positions in the proofreading text and the error correction text is very small, and one editing operation or no editing operation is required to obtain the same character length (similarity). It also proves that there are recognition errors in the character strings of the recognized text at different positions. According to a preset marking method, the character strings with recognition errors are positioned and marked in the proofreading text and the recognized text (for example, the characters with recognition errors are made bolder in the proofreading text and the recognized text).
[0089] When the second editing number is greater than the preset value, it means that the similarities between the character strings at different positions in the proofreading text and the error correction text are very different, and multiple editing operations are required to obtain the same character length. It also means that the character strings of the recognition text, error correction text and proofreading text at different positions are different, indicating that the image text input by the user may have been modified. According to the preset marking method, the character string that may have been modified is positioned and marked in the proofreading text and the recognition text (for example, the content that may have been modified is highlighted in other colors or underlined in the proofreading text and the recognition text), reminding the user to focus on checking the content of the positioning mark.
[0090] In one embodiment, the positioning and marking the character strings at different positions in the proofreading text and the recognition text includes:
[0091] When the second editing number is less than or equal to the preset value, the character strings at different positions in the proofreading text and the recognition text are first marked according to a preset marking method;
[0092] When the second editing times is greater than the preset value, the character strings at different positions are second marked in the proofreading text and the recognition text according to a preset marking method.
[0093] The first mark makes the character string bold, highlights it with a light color, etc., and the second mark makes the character string dark, underlined, shadowed or glowing, etc. According to different marks, the user clearly knows what problems have occurred in the marked content in the proofreading text and the recognized text, so as to quickly find out what content has been changed from the picture text.
[0094] Another situation that may occur in the present invention is: when the character strings of the recognition text, error correction text and proofreading text at the same position are different, if the user verifies the image text and finds that the content of the image text and the proofreading text at the same position is the same and has not been modified, it means that the character strings of the recognition text and the error correction text are both wrong, which also means that the text error correction model is not ideal. It is necessary to optimize the text error correction pre-training model or add other text error correction models to achieve the purpose of integrated learning to improve the accuracy of text error correction. However, the introduction of other text error correction models will also increase the amount of calculation and the reasoning time.
[0095] Reference Figure 2 , which is a schematic diagram of the functional modules of the text comparison device 100 of the present invention.
[0096] The text comparison device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the text comparison device 100 can include an error correction module 110, a calculation module 120, an output module 130 and a feedback module 140. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.
[0097] In this embodiment, the functions of each module / unit are as follows:
[0098] Error correction module: used for receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain corrected text;
[0099] A calculation module: used for calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain a first edit number of character strings at different positions in the proofread text and the recognized text;
[0100] Output module: used for calculating the second editing times of the strings of the proofreading text and the error correction text at the different positions according to a second preset editing distance formula when the first editing times of the strings at the different positions are greater than a preset value;
[0101] Feedback module: used to determine whether the character strings at different positions in the recognized text are modified content or incorrectly recognized content based on the size relationship between the second number of edits and the preset value, mark the character strings at different positions in the proofreading text and the recognized text, and feed back the marked proofreading text and the recognized text to the user.
[0102] In one embodiment, the preset language model group includes a first pre-trained language model and a second pre-trained language model, the first pre-trained language model is a BERT4CSC pre-trained language model, the second pre-trained language model is a MacBERT pre-trained language model, and the recognition text is input into the preset language model group for error correction to obtain the error correction text, including:
[0103] Input the recognition text into the first pre-trained language model and the second pre-trained language model respectively for word segmentation to obtain a plurality of words, and calculate each word according to a preset error correction text formula of the preset language model group to obtain a target probability of all predicted words corresponding to each word;
[0104] A predicted word with the maximum target probability is selected from all predicted words corresponding to each word, and each word is replaced according to the predicted word with the maximum target probability to obtain the error correction text.
[0105] In one embodiment, the preset error correction text formula includes:
[0106] +
[0107] in, is the error correction text, for the recognized text in the first pre-trained language model, For the recognized text in the second pre-trained language model, , is the weighting coefficient, and softmax is the normalization function.
[0108] In one embodiment, before calculating the proofreading text and the recognized text according to the first preset edit distance formula, the method further includes:
[0109] All special characters of the proofreading text, the recognition text and the error correction text are standardized in a unified format according to a pre-constructed special character dictionary.
[0110] In one embodiment, the calculating the second edit times of the character strings of the proofreading text and the error correction text at the different positions according to the second preset edit distance formula includes:
[0111] Read the character strings of the proofreading text and the error correction text at different positions to compare their lengths;
[0112] Calculating the length of the character string of the error correction text according to the second preset edit distance formula and transforming it into the length of the character string of the proofreading text to obtain the number of edits for different operation modes;
[0113] The minimum number of edits among the different operation modes is selected as the second number of edits of the character strings of the proofreading text and the error correction text at the different positions.
[0114] In one embodiment, the second preset edit distance formula includes:
[0115]
[0116] in, The proofread text The first character of the recognition text and the first character of the error correction text The number of edits of characters, The proofread text characters, The first characters, The first characters, This is the second conditional statement.
[0117] In one embodiment, the positioning and marking the character strings at different positions in the proofreading text and the recognition text includes:
[0118] When the second editing number is less than or equal to the preset value, the character strings at different positions in the proofreading text and the recognition text are first marked according to a preset marking method;
[0119] When the second editing times is greater than the preset value, the character strings at different positions are second marked in the proofreading text and the recognition text according to a preset marking method.
[0120] Reference Figure 3 FIG. 1 is a schematic diagram of a preferred embodiment of an electronic device 1 of the present invention.
[0121] The electronic device 1 includes but is not limited to: a memory 11, a processor 12, a display 13 and a network interface 14. The electronic device 1 is connected to a network through the network interface 14 to obtain raw data. The network may be a wireless or wired network such as an intranet, the Internet, a global system of mobile communication (GSM), a wideband code division multiple access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, a call network, etc.
[0122] Among them, the memory 11 includes at least one type of readable medium, and the readable medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a hard disk or memory of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped with the electronic device 1. Of course, the memory 11 can also include both the internal storage unit of the electronic device 1 and its external storage device. In this embodiment, the memory 11 is generally used to store the operating system and various application software installed on the electronic device 1, such as the program code of the text comparison program 10, etc. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.
[0123] The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the program code of the text comparison program 10.
[0124] The display 13 may be referred to as a display screen or a display unit. In some embodiments, the display 13 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (OLED) touch device. The display 13 is used to display information processed in the electronic device 1 and to display a visual working interface, such as displaying the results of data statistics.
[0125] The network interface 14 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface). The network interface 14 is generally used to establish a communication connection between the electronic device 1 and other electronic devices.
[0126] Figure 3Only the electronic device 1 having the components 11 - 14 and the text comparison program 10 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0127] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (Organic Light-Emitting Diode, OLED) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.
[0128] The electronic device 1 may further include a radio frequency (RF) circuit, a sensor, an audio circuit, etc., which will not be described in detail here.
[0129] In the above embodiment, the processor 12 can implement the following steps when executing the text comparison program 10 stored in the memory 11:
[0130] Receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain error-corrected text;
[0131] Calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain first edit times of character strings at different positions in the proofread text and the recognized text;
[0132] When the first editing times of the character strings at different positions are greater than a preset value, calculating the second editing times of the character strings at different positions of the proofreading text and the error correction text according to a second preset editing distance formula;
[0133] Based on the relationship between the second number of edits and the preset value, it is determined that the character strings at different positions in the recognized text are modified content or incorrectly recognized content, and the character strings at different positions are positioned and marked in the proofreading text and the recognized text, and the proofreading text and the recognized text with the positioning marks are fed back to the user.
[0134] The storage device may be the memory 11 of the electronic device 1 , or may be another storage device that is communicatively connected to the electronic device 1 .
[0135] For a detailed description of the above steps, please refer to the above Figure 2 Functional module diagram of the text comparison device 100 embodiment and Figure 1 Description of a flowchart of an embodiment of a text comparison method.
[0136] In addition, an embodiment of the present invention further proposes a computer-readable medium, which may be non-volatile or volatile. The computer-readable medium may be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable medium includes a data storage area and a program storage area. The data storage area stores data created according to the use of the blockchain node, and the program storage area stores a text comparison program 10. When the text comparison program 10 is executed by the processor, the following operations are implemented:
[0137] Receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain error-corrected text;
[0138] Calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain first edit times of character strings at different positions in the proofread text and the recognized text;
[0139] When the first editing times of the character strings at different positions are greater than a preset value, calculating the second editing times of the character strings at different positions of the proofreading text and the error correction text according to a second preset editing distance formula;
[0140] Based on the relationship between the second number of edits and the preset value, it is determined that the character strings at different positions in the recognized text are modified content or incorrectly recognized content, and the character strings at different positions are positioned and marked in the proofreading text and the recognized text, and the proofreading text and the recognized text with the positioning marks are fed back to the user.
[0141] The specific implementation of the computer-readable medium of the present invention is substantially the same as the specific implementation of the above-mentioned text comparison method, and will not be described in detail here.
[0142] In another embodiment, the text comparison method provided by the present invention can further ensure the privacy and security of all the above data, and all the above data can also be stored in a blockchain node. For example, the first edit number and the second edit number can be stored in the blockchain node.
[0143] It should be noted that the blockchain referred to in the present invention is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.
[0144] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, an electronic device, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0146] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A text comparison method, characterized in that: The method comprises: Receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain error-corrected text; Calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain first edit times of character strings at different positions in the proofread text and the recognized text; When the first editing times of the character strings at different positions are greater than a preset value, calculating the second editing times of the character strings at different positions of the proofreading text and the error correction text according to a second preset editing distance formula; According to the relationship between the second number of edits and the preset value, determining whether the character strings at different positions in the recognized text are modified contents or recognized incorrect contents, marking the character strings at different positions in the proofreading text and the recognized text, and feeding back the marked proofreading text and the recognized text to the user; The preset language model group includes a first pre-trained language model and a second pre-trained language model, the first pre-trained language model is a BERT4CSC pre-trained language model, and the second pre-trained language model is a MacBERT pre-trained language model, and the recognition text is input into the preset language model group for error correction to obtain the error correction text, including: inputting the recognition text into the first pre-trained language model and the second pre-trained language model respectively for word segmentation to obtain a plurality of words, calculating each word according to a preset error correction text formula of the preset language model group to obtain a target probability of all predicted words corresponding to each word; selecting a predicted word with the maximum target probability from all predicted words corresponding to each word, and replacing each word according to the predicted word with the maximum target probability to obtain the error correction text.
2. The text comparison method according to claim 1, characterized in that: The preset error correction text formula includes: + in, is the error correction text, for the recognized text in the first pre-trained language model, For the recognized text in the second pre-trained language model, , is the weighting coefficient, and softmax is the normalization function.
3. The text comparison method according to claim 1, characterized in that: Before calculating the proofreading text and the recognized text according to the first preset edit distance formula, the method further includes: All special characters of the proofreading text, the recognition text and the error correction text are standardized in a unified format according to a pre-constructed special character dictionary.
4. The text comparison method according to claim 1, characterized in that: The calculating, according to the second preset edit distance formula, the second edit times of the character strings of the proofreading text and the error correction text at different positions comprises: Read the character strings of the proofreading text and the error correction text at different positions to compare their lengths; Calculating the length of the character string of the error correction text according to the second preset edit distance formula and transforming it into the length of the character string of the proofreading text to obtain the number of edits for different operation modes; The minimum number of edits among the different operation modes is selected as the second number of edits of the character strings of the proofreading text and the error correction text at the different positions.
5. The text comparison method according to claim 1 or 4, characterized in that: The second preset edit distance formula includes: in, The proofread text The first character of the recognition text and the first character of the error correction text Number of edits of characters, The proofread text characters, The first characters, The first characters, This is the second conditional statement.
6. The text comparison method according to claim 1, characterized in that: The positioning and marking of the character strings at different positions in the proofreading text and the recognition text includes: When the second editing number is less than or equal to the preset value, the character strings at different positions in the proofreading text and the recognition text are first marked according to a preset marking method; When the second editing times is greater than the preset value, the character strings at different positions are second marked in the proofreading text and the recognition text according to a preset marking method.
7. A text comparison device, characterized in that: The device comprises: Error correction module: used for receiving the picture text and proofreading text input by the user, performing optical character recognition on the picture text to obtain recognized text, and then inputting the recognized text into a preset language model group for error correction to obtain corrected text; A calculation module: used for calculating the proofread text and the recognized text according to a first preset edit distance formula to obtain a first edit number of character strings at different positions in the proofread text and the recognized text; Output module: used for calculating the second editing times of the strings of the proofreading text and the error correction text at the different positions according to a second preset editing distance formula when the first editing times of the strings at the different positions are greater than a preset value; Feedback module: used for judging, according to the relationship between the second number of edits and the preset value, whether the character strings at different positions in the recognized text are modified contents or recognized incorrect contents, marking the character strings at different positions in the proofreading text and the recognized text, and feeding back the marked proofreading text and the recognized text to the user; The preset language model group includes a first pre-trained language model and a second pre-trained language model, the first pre-trained language model is a BERT4CSC pre-trained language model, and the second pre-trained language model is a MacBERT pre-trained language model, and the recognition text is input into the preset language model group for error correction to obtain the error correction text, including: inputting the recognition text into the first pre-trained language model and the second pre-trained language model respectively for word segmentation to obtain a plurality of words, calculating each word according to a preset error correction text formula of the preset language model group to obtain a target probability of all predicted words corresponding to each word; selecting a predicted word with the maximum target probability from all predicted words corresponding to each word, and replacing each word according to the predicted word with the maximum target probability to obtain the error correction text.
8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor so that the at least one processor can execute the text comparison method according to any one of claims 1 to 6.
9. A computer-readable medium, characterized in that The computer-readable medium stores a text comparison program, and when the text comparison program is executed by a processor, the steps of the text comparison method according to any one of claims 1 to 6 are implemented.
