Artificial intelligence-based chinese correction method and device, computer device and medium
By using AI-based word segmentation and encoding feature extraction, combined with error detection and type inference models, the replacement probability values of candidate replacement Chinese characters are calculated, which solves the problem of low efficiency of existing Chinese text error correction methods and achieves more efficient and accurate error correction effects.
Patent Information
- Application Number
- CN202411171905.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-24
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2044-08-24
AI Technical Summary
Existing Chinese text error correction methods are inefficient and have poor correction effects in financial enterprises. In particular, the computational complexity is high when processing large-scale databases and long texts, and they are unable to adapt to new vocabulary, resulting in slow system response and inaccurate error correction.
An artificial intelligence-based method is used to calculate the replacement probability values of candidate replacement Chinese characters through word segmentation, coding feature extraction, error detection, type reasoning and prediction models, using pre-trained language models and multi-layer perceptron classifiers, combined with sound-like and shape-like error models, and select the Chinese characters with the highest probability of correction for error correction.
The processing efficiency and accuracy of Chinese text error correction have been improved, and errors in the text can be better identified and corrected, thus improving the error correction effect.
Smart Images

Figure CN119150867B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence development and the field of financial technology, and in particular relates to a Chinese error correction method and device based on artificial intelligence, a computer device and a storage medium. BACKGROUND
[0002] In the daily operation of financial enterprises, ensuring the accuracy of information is the cornerstone of maintaining customer trust, ensuring transaction safety and improving service quality. Especially in scenarios involving a large amount of text data, such as customer data entry, transaction record keeping, financial report preparation, etc., the correctness of the text is particularly important. However, due to human input errors, system recognition biases or language complexity, etc., it is inevitable that there will be errors, typos, etc. in the text data, and if these problems are not corrected in a timely manner, they may cause misunderstandings, data errors and even legal disputes.
[0003] Traditionally, financial enterprises have widely adopted methods based on Pinyin Edit Distance or Keyboard Edit Distance for error correction in Chinese text. The core idea of these methods is to simulate the possible input errors of the user (such as mis-hitting or missing keyboard key positions), calculate the minimum number of editing operations (including insertion, deletion, replacement, etc.) required to convert the incorrect string to the correct string, and thus achieve automatic correction of the incorrect text. Although the error correction method has shown certain effectiveness in specific scenarios, its inherent limitations have become increasingly apparent: 1. High computational complexity: Since each character input needs to be compared with all possible correct strings in the database and the edit distance needs to be calculated, the time complexity of the algorithm is as high as O(N*L), where N represents the total number of characters (i.e. the size of the set of all possible correct strings in the database) and L is the length of the string to be corrected. As the size of the database increases and the length of the text increases, this computational burden will increase dramatically, affecting the response speed and efficiency of the system, resulting in low efficiency of error correction processing. 2. Strong dependence on database content: This type of method is highly dependent on the pre-established correct string database. For words that do not exist in the database or newly emerging words, effective error correction cannot be performed, limiting its scope of application and flexibility, resulting in poor error correction effect. SUMMARY
[0004] The purpose of the embodiments of the present application is to propose a Chinese error correction method, device, computer device and storage medium based on artificial intelligence to solve the technical problems of low efficiency of error correction processing and poor error correction effect of the existing error correction methods for errors in Chinese text in financial enterprises.
[0005] To solve the above technical problems, the embodiment of the present application provides a Chinese error correction method based on artificial intelligence, which adopts the technical scheme as follows:
[0006] An input text to be processed is acquired, and word segmentation processing is performed on the input text to obtain corresponding words;
[0007] The words are subjected to encoding feature extraction processing to obtain corresponding target encoding features;
[0008] The target encoding features are subjected to error detection processing based on a preset error detection model to obtain a corresponding error detection result;
[0009] If the error detection result is a wrong word, the target encoding features are subjected to error type reasoning processing based on a preset error type reasoning model to obtain a corresponding target error type;
[0010] The target encoding features are subjected to prediction processing based on a preset error correction model to obtain a first replacement probability value of each candidate replacement Chinese character corresponding to the target encoding features;
[0011] A target prediction model corresponding to the target error type is called to perform prediction processing on the target encoding features to obtain a second replacement probability value of each candidate replacement Chinese character corresponding to the target encoding features;
[0012] Based on the first replacement probability value and the second replacement probability value, a corrected replacement probability value of each candidate replacement Chinese character corresponding to the target encoding features is generated;
[0013] The target replacement Chinese character with the highest corrected replacement probability value is selected from all the candidate replacement Chinese characters, and the target replacement Chinese character is taken as the corrected word of the word.
[0014] Further, the target error type includes a homophonic error or a homographic error, and the step of calling a target prediction model corresponding to the target error type to perform prediction processing on the target encoding features to obtain a second replacement probability value of each candidate replacement Chinese character corresponding to the target encoding features specifically includes:
[0015] If the target error type is a homophonic error, a preset homophonic model is called as the target prediction model;
[0016] The target encoding features are subjected to prediction processing based on the homophonic model to obtain a first prediction replacement probability value of each candidate replacement Chinese character corresponding to the target encoding features;
[0017] The first prediction replacement probability value is taken as the second replacement probability value;
[0018] If the target error type is a similar error, a preset similar model is called as the target prediction model;
[0019] The target coding feature is predicted based on the similar model to obtain a second predicted replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character;
[0020] The second predicted replacement probability value is taken as the second replacement probability value.
[0021] Further, the step of generating a corrected replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character based on the first replacement probability value and the second replacement probability value specifically includes:
[0022] A first specified replacement probability value and a second specified replacement probability value corresponding to a specified candidate replacement Chinese character are obtained; the specified candidate replacement Chinese character is any one of all the candidate replacement Chinese characters;
[0023] A preset addition strategy is obtained;
[0024] The first specified replacement probability value and the second specified replacement probability value are added based on the addition strategy to obtain a specified replacement probability sum value;
[0025] The specified replacement probability sum value is taken as the corrected replacement probability value of the target coding feature corresponding to the specified candidate replacement Chinese character.
[0026] Further, the step of performing word segmentation processing on the input text to obtain a corresponding word specifically includes:
[0027] A preset word segmentation tool is called;
[0028] The input text is segmented based on the word segmentation tool to obtain a segmentation result;
[0029] The segmentation result is taken as the word.
[0030] Further, the step of performing coding feature extraction processing on the word to obtain a corresponding target coding feature specifically includes:
[0031] A pre-trained language model is called;
[0032] The word is coded based on the pre-trained language model to obtain a first coding feature;
[0033] The first coding feature is regularized to obtain a second coding feature;
[0034] The second encoding feature is taken as the target encoding feature.
[0035] Further, before the step of performing error type inference on the target encoding feature based on a preset error type inference model to obtain a corresponding target error type, the method further comprises:
[0036] obtaining a pre-constructed error word dataset;
[0037] encoding features of the error word dataset to obtain corresponding encoding feature data;
[0038] preprocessing the encoding feature data to obtain corresponding error word sample data;
[0039] calling a preset multi-layer perception classifier and determining a network structure of the multi-layer perception classifier;
[0040] determining a target activation function and a target loss function corresponding to the multi-layer perception classifier;
[0041] training the multi-layer perception classifier using the error word sample data based on the target activation function and the target loss function to obtain a specified model meeting a preset construction condition;
[0042] taking the specified model as the error type inference model.
[0043] Further, after the step of screening a target replacement Chinese character with the highest correction replacement probability value from all the candidate replacement Chinese characters and taking the target replacement Chinese character as the corrected word of the word, the method further comprises:
[0044] performing corresponding replacement processing on the word in the input text based on the corrected word to obtain a corresponding error correction text;
[0045] obtaining a target storage mode corresponding to the error correction text;
[0046] storing the error correction text based on the target storage mode.
[0047] To solve the above technical problems, the embodiment of the present application also provides a Chinese error correction device based on artificial intelligence, which adopts the technical scheme as follows:
[0048] a word segmentation module configured to obtain an input text to be processed and perform word segmentation processing on the input text to obtain corresponding words;
[0049] an extraction module configured to perform encoding feature extraction processing on the words to obtain corresponding target encoding features;
[0050] The detection module is configured to perform error detection processing on the target coding feature based on a preset error detection model to obtain a corresponding error detection result.
[0051] The reasoning module is configured to perform error type reasoning processing on the target coding feature based on a preset error type reasoning model if the error detection result is a wrong word, to obtain a corresponding target error type.
[0052] The first prediction module is configured to perform prediction processing on the target coding feature based on a preset error correction model to obtain a first replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0053] The second prediction module is configured to perform prediction processing on the target coding feature by calling a target prediction model corresponding to the target error type to obtain a second replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0054] The generation module is configured to generate a corrected replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character based on the first replacement probability value and the second replacement probability value.
[0055] The screening module is configured to screen a target replacement Chinese character with the highest corrected replacement probability value from all the candidate replacement Chinese characters, and take the target replacement Chinese character as a correction word of the word.
[0056] To solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the technical scheme as follows:
[0057] The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the Chinese error correction method based on artificial intelligence provided in any one of the embodiments of the present application when executing the computer program.
[0058] To solve the above technical problems, the embodiment of the present application further provides a computer readable storage medium, which adopts the technical scheme as follows:
[0059] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the Chinese error correction method based on artificial intelligence provided in any one of the embodiments of the present application.
[0060] Compared with the prior art, the embodiment of the present application has the following beneficial effects:
[0061] After the input text obtained is segmented and the target coding features are extracted, the error detection model is used to detect the error words in the input text, the target error type of the error word is detected based on the error type reasoning model, the corresponding second replacement probability value is established according to the target error type by using the corresponding target prediction model, and the first specified replacement probability value corresponding to the error word output by the error correction model is subjected to probability correction processing for the target error type, and the corresponding corrected replacement probability value is obtained, so that subsequent word error correction according to the corrected replacement probability value can effectively improve the processing efficiency of the error correction processing, and the judgment of the error word in the text is more accurate, the error correction effect is better, and the error correction effect is effectively improved. BRIEF DESCRIPTION OF DRAWINGS
[0062] In order to more clearly illustrate the scheme in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0063] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;
[0064] Figure 2 Flow chart of an embodiment of the Chinese error correction method based on artificial intelligence according to the present application;
[0065] Figure 3 is a structural schematic diagram of an embodiment of the Chinese error correction device based on artificial intelligence according to the present application;
[0066] Figure 4 is a structural schematic diagram of an embodiment of the computer device according to the present application. DETAILED DESCRIPTION
[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit the present application; the specification, claims and above description of drawings of the present application The terms "include" and "have" and any variations thereof in the specification and claims, are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the specification and claims or above description of drawings are used to distinguish different objects, not to describe a particular order.
[0068] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments.
[0069] For better understanding of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings.
[0070] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, and the like.
[0071] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.
[0072] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and the like.
[0073] The server 105 can be a server providing various services, such as a background server supporting a page displayed on the terminal devices 101, 102, 103.
[0074] It should be noted that the method for correcting Chinese based on artificial intelligence provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the device for correcting Chinese based on artificial intelligence is generally arranged in a server / terminal device.
[0075] It should be understood that Figure 1The number of terminal devices, networks and servers in the above-mentioned system is only illustrative. Any number of terminal devices, networks and servers can be provided according to the implementation needs.
[0076] With reference to the above Figure 2 , a flowchart of one embodiment of the Chinese error correction method based on artificial intelligence according to the present application is shown. The order of the steps in the flowchart can be changed, and some steps can be omitted according to different needs. The Chinese error correction method based on artificial intelligence provided by the embodiments of the present application can be applied to any scenario that needs Chinese error correction, and then the Chinese error correction method based on artificial intelligence can be applied to products in these scenarios, such as Chinese recommendation in the financial insurance field. The Chinese error correction method based on artificial intelligence includes the following steps:
[0077] In step S201, an input text to be processed is obtained, and the input text is subjected to word segmentation processing to obtain corresponding words.
[0078] In the present embodiment, the electronic device (such as the server / terminal device shown in Figure 1 ) on which the Chinese error correction method based on artificial intelligence runs can obtain the input text to be processed through wired connection or wireless connection. It should be noted that the above-mentioned wireless connection can include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection methods. The above-mentioned input text to be processed is a text that needs to be corrected. For example, in the business scenario of Chinese text correction in the financial field, the present application can be applied in the writing of the mail of a financial enterprise, and the content of the mail text can be corrected. It can also be applied in the writing of financial scripts and financial documents of a financial enterprise, and the financial scripts and financial documents can be corrected. It can also be applied to the results of OCR recognition of financial pictures. The specific implementation process of the above-mentioned word segmentation processing of the input text to obtain corresponding words will be further described in detail in the subsequent specific embodiments, and will not be described in detail here.
[0079] In step S202, the words are subjected to encoding feature extraction processing to obtain corresponding target encoding features.
[0080] In the present embodiment, the specific implementation process of the above-mentioned encoding feature extraction processing of the words to obtain corresponding target encoding features will be further described in detail in the subsequent specific embodiments, and will not be described in detail here.
[0081] Step S203, error detection processing is performed on the target coding feature based on a preset error detection model, and a corresponding error detection result is obtained.
[0082] In the embodiment, the error detection model can be a multi-layer perception classifier (first MLP model). The error detection model is composed of a full connection layer and a softmax function, wherein, W D is the full connection layer parameter of the error detection model, d represents the number of dimensions. D is the matrix size, b D is the bias term, p is the probability of the current i-th word, the value of the probability is between 0 and 1, which is a binomial distribution probability, meaning that in order to predict whether the current word is a wrong word, h is the current i-th word, R 2 is the determination coefficient, also known as the coefficient of determination, which is the core index for evaluating the prediction performance of the model. It measures the contribution of the independent variable to the change of the dependent variable. The higher the R 2 value, the stronger the ability of the model to explain the variability of the dependent variable Y, which is the gold standard for measuring the goodness of fit of the model. The item with the maximum value during prediction, represents whether the current i-th word is a wrong word, y=0 represents correct, and y=1 represents error. Since the training data set (after manual annotation) knows which word is wrong, the loss value of detecting the wrong word can be calculated by cross-entropy, and the loss value is gradient backpropagation to train the parameters in the model. The specific training process of the error detection model can refer to the training process of the error correction model below, and only the sample data of the error detection model needs to be replaced by the sample containing correct words and wrong words.
[0083] Step S204, if the error detection result is a wrong word, an error type reasoning model is used to perform error type reasoning processing on the target coding feature, and a corresponding target error type is obtained.
[0084] In the embodiment, the target coding feature is input into the error type inference model, the error type inference model performs error type inference processing on the target coding feature, and the corresponding target error type is output. Specifically, the inference process needs the model to learn the error type of the input text, whether it is a homophone error or a homograph error. Generally, a homophone error is a text error caused by a rolled tongue, and a homograph error is a text error caused by a component. Since it has been detected which word is wrong based on the use of the error detection model, the coding feature of the wrong word is input into another MLP model (second MLP model, i.e., error type inference model). The parameters of the error type inference model are initialized differently from the parameters of the above error detection model, but the structures are consistent, and are also composed of one fully connected layer and one softmax function. wherein, W R is the fully connected layer parameter of the error type inference model, r represents the current vector dimension, and is distinguished from d above. R is the matrix size, which is distinguished from D above, represents the same structure, and the parameter values are different. b R is the bias term. The output p is the error type probability of the coding feature of the word detected as wrong in the detection step, which is also a binomial distribution. The error type is calculated as whether it is a homophone error or a homograph error. Here, y=0 is a homophone, and y=1 is a homograph. Similarly, the error type loss value of the detected wrong word is calculated by cross-entropy. The specific construction process of the error type inference model will be further described in detail in subsequent specific embodiments, and will not be described in detail here.
[0085] In step S205, the target coding feature is predicted based on the preset error correction model to obtain a first replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0086] In the embodiment, the error correction model is an MLP layer with softmax. The MLP layer still has only one fully connected layer. Different from the previous one is not a binomial distribution, but a distribution of all Chinese characters. For example, if there are 13000 Chinese characters, the size of the vocab is 13000. Softmax outputs 13000 probability values, representing the probability that the current wrong word should be replaced by each character. Similarly, the character with the maximum probability value is obtained by argmax to replace the wrong word.
[0087] Since the error type inferred based on the error type inference model should have different effects on the probability of each word in the vocab, the vocab needs to be further modeled for probability correction:
[0088] (1) Manually mark words that are similar in pronunciation, for example, corresponding retroflex and retroflex sounds are grouped into one category (z to zh, c to ch, s to sh), n and l are grouped into one category, g and k are grouped into one category, and front nasal and back nasal sounds are grouped into one category (en to eng, an to ang, on to ong, in to ing, en to on, eng to ong). Then mark words that are similar in form, for example, words that differ by only one stroke, words with a single radical and words with a double radical.
[0089] (2) All words in the vocabulary are paired together to form a word. The words are then fed into two BERT models with completely randomized parameters, one for the sound-similar model and the other for the shape-similar model. Because the words that are similar in sound or shape are annotated, BERT outputs an encoding vector for each word group. The encoding vector for each word group is passed through a d*d fully connected layer and a sigmoid activation function to output a value. All values are normalized using softmax. The normalized value is the replacement probability between the two words. Because there are two BERT models, each group will have two probability values.
[0090] (3) If the error type inferred by the error type inference model is phonetic, the phonetic BERT probability value is added to the predicted value of the replacement word in the third step. If the error type is morphological, the morphological BERT probability value is added to the predicted value of the replacement word obtained based on the error correction model. In the third step, an incorrect word will have a predicted replacement probability value for all words in the vocab. In the probability correction process, an incorrect word will also have a replacement probability value for all words in the vocab. After adding the corresponding probability values, it is the correction value.
[0091] Step S206: calling a target prediction model corresponding to the target error type to perform prediction processing on the target coding feature to obtain a second replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters.
[0092] In this embodiment, the above-mentioned calling of the target prediction model corresponding to the target error type performs prediction processing on the target coding feature to obtain the specific implementation process of the second replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters. This application will provide further detailed descriptions of this in subsequent specific embodiments and will not be elaborated on here.
[0093] Step S207: Based on the first replacement probability value and the second replacement probability value, generate a modified replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters.
[0094] In the embodiment, the specific implementation process of generating the modified replacement probability value corresponding to each candidate replacement Chinese character for the target coding feature based on the first replacement probability value and the second replacement probability value will be further described in details in subsequent embodiments, and will not be elaborated here.
[0095] In step S208, the target replacement Chinese character with the highest modified replacement probability value is screened out from all the candidate replacement Chinese characters, and the target replacement Chinese character is taken as the corrected word of the word.
[0096] In the embodiment, the target modified replacement probability value with the highest value can be screened out by comparing all the modified replacement probability values, and the replacement Chinese character matched with the target modified replacement probability value is extracted from all the candidate replacement Chinese characters to obtain the target replacement Chinese character, and the target replacement Chinese character is taken as the corrected word of the word, which is used for subsequent error correction processing of the word.
[0097] The present application firstly acquires an input text to be processed, and performs word segmentation processing on the input text to obtain corresponding words; and performs encoding feature extraction processing on the words to obtain corresponding target encoding features; then performs error detection processing on the target encoding features based on a preset error detection model to obtain a corresponding error detection result; if the error detection result is a wrong word, performs error type reasoning processing on the target encoding features based on a preset error type reasoning model to obtain a corresponding target error type; then performs prediction processing on the target encoding features based on a preset error correction model to obtain a first replacement probability value of the target encoding features corresponding to each candidate replacement Chinese character; subsequently, a target prediction model corresponding to the target error type is called to perform prediction processing on the target encoding features to obtain a second replacement probability value of the target encoding features corresponding to each candidate replacement Chinese character; further, based on the first replacement probability value and the second replacement probability value, a corrected replacement probability value of the target encoding features corresponding to each candidate replacement Chinese character is generated; finally, a target replacement Chinese character with the highest corrected replacement probability value is selected from all the candidate replacement Chinese characters, and the target replacement Chinese character is taken as a corrected word of the word. After the input text acquired is segmented and the target encoding features are extracted, the error detection model is used to detect a wrong word in the input text, the error type reasoning model is used to detect a target error type of the wrong word, and then the corresponding target prediction model is used to establish a corresponding second replacement probability value according to the target error type, and the first specified replacement probability value corresponding to the wrong word output by the error correction model is subjected to probability correction processing for the target error type to obtain a corresponding corrected replacement probability value, so that subsequent word error correction according to the corrected replacement probability value can effectively improve the processing efficiency of error correction processing, and the judgment of the wrong word in the text is more accurate, the error correction effect is better, and the error correction effect is effectively improved.
[0098] In some optional implementations, the target error type includes a homophonic error or a homographic error; and step S206 includes the following steps:
[0099] If the target error type is a homophonic error, a preset homophonic model is called as the target prediction model.
[0100] In the present embodiment, the construction process of the homophonic model includes: a BERT model is used to encode an annotated homophonic word pair (wrong word-correct word) in advance, and then a model capable of calculating the probability of replacing a given wrong word with a certain homophonic correct word is trained through a full connection layer and a sigmoid function to obtain the above-mentioned homophonic model.
[0101] The target coding feature is predicted based on the phonetic model to obtain a first predicted replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0102] In the embodiment, the target coding feature is input into the phonetic model to perform prediction processing on the target coding feature by the phonetic model, and a second predicted replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character is output.
[0103] The first predicted replacement probability value is taken as the second replacement probability value.
[0104] If the target error type is a shape-similarity error, a preset shape-similarity model is called as the target prediction model.
[0105] In the embodiment, the construction process of the shape-similarity model includes: a BERT model is used to encode a labeled shape-similarity word pair (wrong word-correct word) in advance, and then a model capable of calculating the probability of replacing a given wrong word with a shape-similarity correct word is trained through a full connection layer and a sigmoid function, to obtain the phonetic model.
[0106] The target coding feature is predicted based on the shape-similarity model to obtain a second predicted replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0107] In the embodiment, the target coding feature is input into the phonetic model to perform prediction processing on the target coding feature by the phonetic model, and a second predicted replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character is output.
[0108] The second predicted replacement probability value is taken as the second replacement probability value.
[0109] After the target error type is determined, the target prediction model matched with the target error type is intelligently called to perform corresponding prediction processing on the target coding feature, so that the second replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character can be accurately obtained, and the data accuracy of the obtained second replacement probability value is effectively ensured.
[0110] In some optional implementation manners of the embodiment, the step S207 includes the following steps.
[0111] The first specified replacement probability value and the second specified replacement probability value corresponding to the specified candidate replacement Chinese character are obtained.
[0112] In the embodiment, the specified candidate replacement Chinese character is any one of all the candidate replacement Chinese characters. The first specified replacement probability value can be obtained by obtaining a first replacement probability value corresponding to the specified candidate replacement Chinese character in the target coding feature, and the second specified replacement probability value can be obtained by obtaining a second replacement probability value corresponding to the specified candidate replacement Chinese character in the target coding feature.
[0113] A preset addition strategy is obtained.
[0114] In the embodiment, the addition strategy can directly use simple addition, or can use weighted addition. The model weight can be set according to the contribution of two models.
[0115] Based on the addition strategy, the first specified replacement probability value and the second specified replacement probability value are added to obtain a corresponding specified replacement probability sum value.
[0116] In the embodiment, the first specified replacement probability value and the second specified replacement probability value are added by using the addition mode corresponding to the addition strategy, so as to obtain the corresponding specified replacement probability sum value.
[0117] The specified replacement probability sum value is used as a corrected replacement probability value corresponding to the specified candidate replacement Chinese character in the target coding feature.
[0118] The application obtains a preset addition strategy, then based on the addition strategy, the first specified replacement probability value and the second specified replacement probability value are added to obtain a corresponding specified replacement probability sum value, and subsequently the specified replacement probability sum value is used as a corrected replacement probability value corresponding to the specified candidate replacement Chinese character in the target coding feature. By using the addition strategy, the first specified replacement probability value and the second specified replacement probability value corresponding to the specified candidate replacement Chinese character are added to correct the probability of the error type and obtain the corresponding corrected replacement probability value, so that the judgment of the error word in the text is more accurate according to the corrected replacement probability value, and the error correction effect is effectively improved.
[0119] In some optional implementation manners, the input text is segmented to obtain corresponding words in step S201, including the following steps:
[0120] A preset segmentation tool is called.
[0121] In the embodiment, the selection of the segmentation tool is not limited, and can be determined according to actual use requirements, for example, HanLP, jieba or other segmentation tools can be selected.
[0122] The input text is processed by using the word segmentation tool to obtain a corresponding word segmentation result.
[0123] In this embodiment, the input text is processed by using the word segmentation tool to obtain a corresponding word segmentation result. For example, the input text X is processed by using the word segmentation tool to obtain a corresponding word sequence [x1, x2,..., xn].
[0124] The word segmentation result is taken as the word.
[0125] The preset word segmentation tool is called, the input text is processed by using the word segmentation tool to obtain a corresponding word segmentation result, and the word segmentation result is taken as the word. By using the word segmentation tool, the word segmentation of the input text can be quickly and accurately completed, and the accuracy of the obtained word is ensured.
[0126] In some optional implementations, step S202 includes the following steps:
[0127] A pre-trained language model is called.
[0128] In this embodiment, the pre-trained language model can be a BERT or GPT model.
[0129] The word is encoded by using the pre-trained language model to obtain a corresponding first encoding feature.
[0130] In this embodiment, the word is input into the pre-trained language model after word segmentation, the word is encoded by using the pre-trained language model to obtain an encoding feature vector of each word, that is, the first encoding feature. For example, the encoding feature of the first word is h1, the encoding feature of the second word is h2, and so on, and the encoding feature set of the entire input text is [h1, h2,..., hn].
[0131] The first encoding feature is regularized to obtain a corresponding second encoding feature.
[0132] In this embodiment, the first encoding features are regularized, that is, the mean and variance of all the first encoding features are calculated, all the first encoding features are subtracted by the mean and then divided by the variance, and the corresponding second encoding features are obtained. By performing the regularization, all the feature values are coordinate-centered and are in the same order of magnitude, which facilitates the training of the model parameters through the loss gradient and facilitates subsequent processing.
[0133] The second encoding feature is taken as the target encoding feature.
[0134] The application encodes the words based on the pre-trained language model to obtain corresponding first encoding features, then performs regularization processing on the first encoding features to obtain corresponding second encoding features, and subsequently takes the second encoding features as the target encoding features. The application encodes the words based on the use of the pre-trained language model to obtain corresponding first encoding features, and then performs regularization processing on the first encoding features, so as to accurately extract the encoding features of the words, and ensure the data specification and accuracy of the target encoding features.
[0135] In some optional implementations of the embodiment, before step S204, the electronic device can further perform the following steps:
[0136] Obtain a pre-constructed wrong word data set.
[0137] In the embodiment, a wrong word data set containing homophonic errors and homographic errors is pre-collected. The wrong word data set contains a sufficient number of samples and covers different types of error cases to ensure the generalization ability of the model.
[0138] Feature encode the wrong word data set to obtain corresponding encoding feature data.
[0139] In the embodiment, for the wrong word data, feature encoding processing is performed to convert the original wrong word data into a numerical form that can be processed by the model. The feature encoding method can include word embedding (such as Word2Vec, GloVe, etc.), character-level encoding, or specific encoding based on error types. In addition, for homophonic errors and homographic errors, different features need to be designed to capture the unique properties of the two types of errors.
[0140] Preprocess the encoding feature data to obtain corresponding wrong word sample data.
[0141] In the embodiment, the preprocessing includes data cleaning, feature selection, and feature normalization processing. Data cleaning includes removing noise, handling missing values, and the like. Feature selection refers to selecting features useful for classification tasks. Feature normalization processing is used to ensure that different features are in the same order of magnitude numerically so that the model can learn.
[0142] Call a preset multi-layer perception classifier and determine the network structure of the multi-layer perception classifier.
[0143] In the embodiment, the multi-layer perception classifier (MLP model) is composed of one fully connected layer and a softmax function.
[0144] determine a target activation function and a target loss function corresponding to the multi-layer perceptron classifier.
[0145] In the embodiment, the target activation function can be specifically a softmax function. The target loss function can be specifically a cross-entropy loss function, which is used to measure the difference between the model prediction result and the actual label.
[0146] Based on the target activation function and the target loss function, the multi-layer perceptron classifier is trained using the wrong word sample data to obtain a specified model meeting a preset construction condition.
[0147] In the embodiment, the construction process of the specified model includes: (1) a training process. The preprocessed wrong word sample data is input into the MLP model, the prediction result is calculated through forward propagation, then the gradient of the loss function is calculated through the back propagation algorithm, and the weights and biases of the model are updated. A plurality of training cycles are repeatedly performed until the performance of the model reaches a satisfactory level or reaches a preset number of training times. (2) Overfitting prevention. During the training process, attention should be paid to preventing overfitting. The risk of overfitting can be reduced by adding a regularization term (such as L1 regularization, L2 regularization), using the Dropout technology or adjusting the learning rate, etc. (3) Model evaluation. The trained MLP model is evaluated using an independent test set. The accuracy, recall rate, F1 score and other indicators of the model are calculated to evaluate the performance of the model in the homophonic error and homographic error classification task. (4) Model tuning. The model is tuned according to the evaluation results. It can include adjusting the network structure (such as increasing or decreasing the number of hidden layers, the number of nodes), replacing the activation function, adjusting the learning rate and other parameters to further improve the performance of the model. Through the above steps, an MLP classifier capable of judging whether a wrong word is a homophonic error or a homographic error according to the encoding features of the wrong word can be trained.
[0148] The specified model is used as the error type reasoning model.
[0149] The present application encodes and preprocesses the pre-constructed wrong word data set to obtain wrong word sample data, and then trains the multi-layer perceptron classifier using the wrong word sample data based on the determined target activation function and target loss function corresponding to the multi-layer perceptron classifier, thereby quickly and intelligently constructing an error type reasoning model meeting the preset construction condition, improving the construction efficiency of the error type reasoning model, and ensuring the model effect of the error type reasoning model.
[0150] In some optional implementation manners of the embodiment, after step S208, the electronic device can further perform the following steps:
[0151] replace the word in the input text with the correction word to obtain a corresponding error-corrected text.
[0152] In the embodiment, the error-corrected text is obtained by determining the position of the word corresponding to the word in the input text and replacing the word at the position with the correction word.
[0153] A target storage mode corresponding to the error-corrected text is obtained.
[0154] In the embodiment, the selection of the target storage mode is not specifically limited and can be determined according to actual business storage requirements. For example, the target storage mode can be any one of cloud server storage, local database storage, network disk storage, and blockchain storage. Preferably, the blockchain storage is used as the target storage mode. The use of the blockchain to store and manage the error-corrected text can effectively ensure the security and tamper resistance of the error-corrected text.
[0155] The error-corrected text is stored based on the target storage mode.
[0156] In the embodiment, the error-corrected text can be stored in a target storage medium corresponding to the target storage mode.
[0157] The application obtains the error-corrected text by replacing the word in the input text with the correction word, obtains a target storage mode corresponding to the error-corrected text, and stores the error-corrected text based on the target storage mode. After the target replacement Chinese character with the highest correction replacement probability value is selected from all candidate replacement Chinese characters as the correction word, the input text is automatically and intelligently replaced with the correction word to obtain the error-corrected text, which ensures the data accuracy of the error-corrected text. In addition, the error-corrected text is further stored based on the target storage mode corresponding to the error-corrected text, which effectively ensures the data security and tamper resistance of the error-corrected text and improves the storage intelligence of the error-corrected text.
[0158] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.
[0159] It is emphasized that, in order to further ensure the privacy and security of the above-mentioned correction words, the above-mentioned correction words can also be stored in a node of a block chain.
[0160] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing a batch of network transaction information for verifying the validity (anti-fake) of the information and generating the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.
[0161] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, methods, technologies and application systems.
[0162] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.
[0163] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, the storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) and other non-volatile storage media, or a random access memory (RAM) and the like.
[0164] It should be understood that, although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.
[0165] Further referring to Figure 3 , as an implementation of the method shown in the above Figure 2 , the present application provides an embodiment of a Chinese error correction device based on artificial intelligence. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.
[0166] As shown in Figure 3 , the Chinese error correction device based on artificial intelligence 300 described in the embodiment includes a word segmentation module 301, an extraction module 302, a detection module 303, an inference module 304, a first prediction module 305, a second prediction module 306, a generation module 307, and a screening module 308. Among them:
[0167] The word segmentation module 301 is configured to obtain an input text to be processed, and perform word segmentation processing on the input text to obtain corresponding words;
[0168] The extraction module 302 is configured to perform coding feature extraction processing on the words to obtain corresponding target coding features;
[0169] The detection module 303 is configured to perform error detection processing on the target coding features based on a preset error detection model to obtain a corresponding error detection result;
[0170] The inference module 304 is configured to, if the error detection result is a wrong word, perform error type inference processing on the target coding features based on a preset error type inference model to obtain a corresponding target error type;
[0171] The first prediction module 305 is configured to perform prediction processing on the target coding features based on a preset error correction model to obtain a first replacement probability value of each candidate replacement Chinese character corresponding to the target coding features;
[0172] The second prediction module 306 is configured to call a target prediction model corresponding to the target error type to perform prediction processing on the target coding feature, so as to obtain a second replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0173] The generation module 307 is configured to generate a corrected replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character based on the first replacement probability value and the second replacement probability value.
[0174] The screening module 308 is configured to screen a target replacement Chinese character with the highest corrected replacement probability value from all the candidate replacement Chinese characters, and take the target replacement Chinese character as the corrected word of the word.
[0175] In the embodiment, the above modules or units are respectively used to perform operations corresponding to the steps of the Chinese error correction method based on artificial intelligence in the foregoing embodiments, and thus will not be described here.
[0176] In some optional implementations of the embodiment, the target error type includes a homophonic error or a homographic error; and the second prediction module 306 includes:
[0177] The first calling sub-module is configured to call a preset homophonic model as the target prediction model if the target error type is a homophonic error.
[0178] The first prediction sub-module is configured to perform prediction processing on the target coding feature based on the homophonic model, so as to obtain a first prediction replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0179] The first determination sub-module is configured to take the first prediction replacement probability value as the second replacement probability value.
[0180] The second calling sub-module is configured to call a preset homographic model as the target prediction model if the target error type is a homographic error.
[0181] The second prediction sub-module is configured to perform prediction processing on the target coding feature based on the homographic model, so as to obtain a second prediction replacement probability value of the target coding feature corresponding to each candidate replacement Chinese character.
[0182] The second determination sub-module is configured to take the second prediction replacement probability value as the second replacement probability value.
[0183] In some optional implementations of the embodiment, the generation module 307 includes:
[0184] The first obtaining sub-module is configured to obtain a first specified replacement probability value and a second specified replacement probability value corresponding to a specified candidate replacement Chinese character; the specified candidate replacement Chinese character is any one of all the candidate replacement Chinese characters;
[0185] The second obtaining sub-module is configured to obtain a preset addition strategy.
[0186] The addition sub-module is configured to add the first specified replacement probability value and the second specified replacement probability value based on the addition strategy to obtain a specified replacement probability sum value.
[0187] The third determining sub-module is configured to take the specified replacement probability sum value as a modified replacement probability value corresponding to the specified candidate replacement Chinese character of the target encoding feature.
[0188] In some optional implementations of the embodiment, the word segmentation module 301 includes:
[0189] The third calling sub-module is configured to call a preset word segmentation tool.
[0190] The word segmentation sub-module is configured to perform word segmentation processing on the input text based on the word segmentation tool to obtain a corresponding word segmentation result.
[0191] The fourth determining sub-module is configured to take the word segmentation result as the word.
[0192] In some optional implementations of the embodiment, the extraction module 302 includes:
[0193] The fourth calling sub-module is configured to call a pre-trained language model.
[0194] The encoding sub-module is configured to perform encoding processing on the word based on the pre-trained language model to obtain a corresponding first encoding feature.
[0195] The regularization sub-module is configured to perform regularization processing on the first encoding feature to obtain a corresponding second encoding feature.
[0196] The fifth determining sub-module is configured to take the second encoding feature as the target encoding feature.
[0197] In some optional implementations of the embodiment, the Chinese error correction apparatus based on artificial intelligence further includes:
[0198] The first obtaining module is configured to obtain a pre-constructed error word data set.
[0199] The encoding module is configured to perform feature encoding on the error word data set to obtain corresponding encoding feature data.
[0200] A preprocessing module is configured to preprocess the encoded feature data to obtain corresponding error word sample data.
[0201] A calling module is configured to call a preset multi-layer perception classifier and determine a network structure of the multi-layer perception classifier.
[0202] A first determining module is configured to determine a target activation function and a target loss function corresponding to the multi-layer perception classifier.
[0203] A training module is configured to train the multi-layer perception classifier based on the target activation function and the target loss function using the error word sample data to obtain a specified model meeting a preset construction condition.
[0204] A second determining module is configured to use the specified model as the error type reasoning model.
[0205] In some optional implementations of the embodiment, the Chinese error correction device based on artificial intelligence further includes:
[0206] A replacing module is configured to replace the word in the input text based on the corrected word to obtain a corresponding error correction text.
[0207] A second obtaining module is configured to obtain a target storage mode corresponding to the error correction text.
[0208] A storage module is configured to store the error correction text based on the target storage mode.
[0209] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the embodiment is shown in the following figure.
[0210] The computer device 4 includes a memory 41, a processor 42, and a network interface 43, which are communicatively connected by a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0211] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and the like.
[0212] The memory 41 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In the present embodiment, the memory 41 is generally used to store an operating system and various application software installed in the computer device 4, such as computer readable instructions of the artificial intelligence-based Chinese error correction method, and the like. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.
[0213] The processor 42 may, in some embodiments, be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In the present embodiment, the processor 42 is configured to execute computer-readable instructions stored in the memory 41 or to process data, such as computer-readable instructions of the artificial intelligence-based Chinese error correction method.
[0214] The network interface 43 may include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 4 and other electronic devices.
[0215] The present application also provides another embodiment, i.e., a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based Chinese error correction method as described above.
[0216] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software and a general hardware platform, and of course, can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to perform the methods described in the various embodiments of the present application.
[0217] Obviously, the above-described embodiments are only some of the embodiments of the present application, rather than all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by referring to the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.
Claims
1. A Chinese error correction method based on artificial intelligence, characterized in that: The steps include: Obtaining an input text to be processed, and performing word segmentation on the input text to obtain corresponding words; Performing coding feature extraction on the words to obtain corresponding target coding features; Performing error detection processing on the target coding feature based on a preset error detection model to obtain a corresponding error detection result; If the error detection result is an incorrect word, error type inference processing is performed on the target coding feature based on a preset error type inference model to obtain a corresponding target error type; Performing prediction processing on the target coding feature based on a preset error correction model to obtain a first replacement probability value of each preset candidate replacement Chinese character corresponding to the target coding feature; Calling a target prediction model corresponding to the target error type to perform prediction processing on the target coding feature to obtain a second replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters; Based on the first replacement probability value and the second replacement probability value, generating a modified replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters; A target replacement Chinese character with the highest correction replacement probability value is selected from all the candidate replacement Chinese characters, and the target replacement Chinese character is used as the correction word of the word.
2. The Chinese error correction method based on artificial intelligence according to claim 1 is characterized in that: The target error type includes a sound-like error or a shape-like error; and the step of calling a target prediction model corresponding to the target error type to perform prediction processing on the target coding feature to obtain a second replacement probability value corresponding to each candidate replacement Chinese character of the target coding feature specifically includes: If the target error type is a sound-like error, calling a preset sound-like model as the target prediction model; Performing prediction processing on the target coding feature based on the sound-like model to obtain a first predicted replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters; Using the first predicted replacement probability value as the second replacement probability value; If the target error type is a similarity error, calling a preset similarity model as the target prediction model; Performing prediction processing on the target coding feature based on the similarity model to obtain a second predicted replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters; The second predicted replacement probability value is used as the second replacement probability value.
3. The Chinese error correction method based on artificial intelligence according to claim 1 is characterized in that: The step of generating, based on the first replacement probability value and the second replacement probability value, a modified replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters specifically includes: Obtaining a first designated replacement probability value and a second designated replacement probability value corresponding to a designated candidate replacement Chinese character; wherein the designated candidate replacement Chinese character is any one of all the candidate replacement Chinese characters; Get the preset addition strategy; Based on the addition strategy, the first designated replacement probability value and the second designated replacement probability value are added to obtain a corresponding designated replacement probability sum value; The specified replacement probability and value are used as the modified replacement probability value of the target coding feature corresponding to the specified candidate replacement Chinese character.
4. The Chinese error correction method based on artificial intelligence according to claim 1 is characterized in that: The step of performing word segmentation on the input text to obtain corresponding words specifically includes: Call the preset word segmentation tool; Perform word segmentation processing on the input text based on the word segmentation tool to obtain corresponding word segmentation results; The word segmentation result is used as the word.
5. The Chinese error correction method based on artificial intelligence according to claim 1 is characterized in that: The step of performing coding feature extraction processing on the words to obtain corresponding target coding features specifically includes: Call the pre-trained language model; Encoding the word based on the pre-trained language model to obtain a corresponding first encoding feature; Performing regularization processing on the first coding feature to obtain a corresponding second coding feature; The second coding feature is used as the target coding feature.
6. The Chinese error correction method based on artificial intelligence according to claim 1 is characterized in that: Before the step of performing error type inference processing on the target coding feature based on a preset error type inference model to obtain a corresponding target error type, the method further includes: Get a pre-built dataset of incorrect words; Performing feature encoding on the incorrect word dataset to obtain corresponding encoded feature data; Preprocessing the encoded feature data to obtain corresponding incorrect word sample data; Calling a preset multi-layer perceptron classifier and determining the network structure of the multi-layer perceptron classifier; Determining a target activation function and a target loss function corresponding to the multilayer perceptron classifier; Based on the target activation function and the target loss function, the multi-layer perceptron classifier is trained using the incorrect word sample data to obtain a specified model that meets the preset construction conditions; The specified model is used as the error type inference model.
7. The Chinese error correction method based on artificial intelligence according to claim 1 is characterized in that: After the step of selecting a target replacement Chinese character with the highest probability of corrected replacement from all the candidate replacement Chinese characters and using the target replacement Chinese character as the corrected word for the phrase, the method further includes: Perform corresponding replacement processing on the words in the input text based on the correction words to obtain a corresponding error-corrected text; Obtaining a target storage mode corresponding to the error correction text; The error correction text is stored based on the target storage method.
8. A Chinese error correction device based on artificial intelligence, characterized in that: include: A word segmentation module is used to obtain an input text to be processed and perform word segmentation on the input text to obtain corresponding words; An extraction module is used to perform coding feature extraction processing on the words to obtain corresponding target coding features; A detection module is used to perform error detection processing on the target coding feature based on a preset error detection model to obtain a corresponding error detection result; An inference module, configured to perform error type inference processing on the target coding feature based on a preset error type inference model to obtain a corresponding target error type if the error detection result is an incorrect word; a first prediction module, configured to perform prediction processing on the target coding feature based on a preset error correction model to obtain a first replacement probability value of each preset candidate replacement Chinese character corresponding to the target coding feature; A second prediction module is configured to call a target prediction model corresponding to the target error type to perform prediction processing on the target coding feature, and obtain a second replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters; a generating module, configured to generate, based on the first replacement probability value and the second replacement probability value, a modified replacement probability value of the target coding feature corresponding to each of the candidate replacement Chinese characters; The screening module is used to screen out the target replacement Chinese character with the highest correction replacement probability value from all the candidate replacement Chinese characters, and use the target replacement Chinese character as the correction word of the word.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the Chinese error correction method based on artificial intelligence as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the Chinese error correction method based on artificial intelligence as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Text information processing method and device, equipment and storage medium
CN116956888A
Chinese spelling error correction method suitable for medical context
CN117648923A