Text punctuation correction method, device, electronic device and storage medium

Abstract

The present application is applicable to the fields of text processing and intelligent decision-making technology, and provides a method, device, electronic device and storage medium for text punctuation correction, the method comprising: obtaining target text content, the target text content being the text content that needs to be corrected by punctuation input by the user; using a pre-configured error detection rule engine and a pre-trained error detection model to perform punctuation error detection on the target text content, respectively, and obtaining a first detection result and a second detection result accordingly; obtaining a list of incorrect punctuation corresponding to the target text content by comparing and merging the first detection result with the second detection result, and performing punctuation correction on the target text content according to the incorrect punctuation list. The method performs error detection and correction on punctuation in the target text content by integrating the rule engine and the deep learning model, and has strong applicability and high accuracy.

Description

Technical Field

[0001] The present application relates to the field of text processing and intelligent decision-making technology, and in particular to a text punctuation correction method, device, electronic device and storage medium. Background Art

[0002] In an article, correct punctuation can help users understand the article correctly, while incorrect punctuation may prevent users from correctly understanding the specific meaning of the article. Therefore, in an article, the correctness of the punctuation is of great significance to the understanding of the article. For the prediction of punctuation errors in articles, a rule engine based on manual settings is usually used for matching and investigation. However, the correction effects of rule engines in different usage scenarios are often uneven, with poor scenario applicability and low accuracy. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a text punctuation correction method, device, electronic device and storage medium, which can improve the accuracy of predicting and correcting punctuation marks in articles.

[0004] A first aspect of an embodiment of the present application provides a text punctuation correction method, comprising:

[0005] Acquire target text content, where the target text content is text content input by a user that needs to be punctuation corrected;

[0006] Using a pre-configured error detection rule engine and a pre-trained error detection model to perform punctuation error detection on the target text content, respectively, to obtain a first detection result and a second detection result;

[0007] By comparing and merging the first detection result and the second detection result, a list of incorrect punctuation corresponding to the target text content is obtained, and punctuation correction processing is performed on the target text content according to the list of incorrect punctuation.

[0008] In conjunction with the first aspect, in a first possible implementation manner of the first aspect, the step of performing punctuation error detection on the target text content using a preconfigured error detection rule engine includes:

[0009] Obtaining punctuation to be detected in the target text content;

[0010] Traversing a first error type table based on a general standard punctuation configuration according to the punctuation to be detected, and determining whether the punctuation to be detected belongs to at least one error type in the first error type table; and / or traversing a second error type table based on a text punctuation collocation configuration according to the punctuation to be detected, and determining whether the punctuation to be detected belongs to at least one error type in the second error type table;

[0011] If the punctuation mark to be detected belongs to at least one error type in the first error type table and / or the punctuation mark to be detected belongs to at least one error type in the second error type table, the punctuation mark to be detected is marked as an erroneous punctuation mark in the first detection result.

[0012] In combination with the first aspect or the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the step of using a pre-trained error detection model to perform punctuation error detection on the target text content includes:

[0013] Decomposing the target text content into non-punctuation segments according to punctuation marks;

[0014] Obtaining a punctuation mark to be detected in the target text content, assembling two punctuation-free sentence segments before and after the punctuation mark to be detected, and obtaining a sentence to be detected for detecting the punctuation mark to be detected;

[0015] Using a pre-trained embedding model to vectorize the obtained sentence to be detected, so as to obtain a vector matrix for representing the sentence to be detected;

[0016] Using a pre-trained bidirectional long short-term memory network model to perform classification prediction processing on the vector matrix, so as to obtain a predicted punctuation type corresponding to the punctuation to be detected based on the sentence to be detected;

[0017] The predicted punctuation type is compared with the punctuation to be detected. If the punctuation to be detected is inconsistent with the predicted punctuation type, the punctuation to be detected is marked as an erroneous punctuation in the second detection result.

[0018] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, before the step of using a pre-trained embedding model to vectorize the obtained sentence to be detected to obtain a vector matrix for characterizing the sentence to be detected, the method further includes:

[0019] Identifying the sequence length of the sentence to be detected;

[0020] Comparing the sequence length of the sentence to be detected with a preset sequence length threshold to obtain a comparison result;

[0021] If the comparison result is that the sequence length of the sentence to be detected is less than the preset sequence length threshold, padding the sentence to be detected so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold;

[0022] If the comparison result is that the sequence length of the sentence to be detected is greater than the preset sequence length threshold, the sentence to be detected is truncated so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold.

[0023] In combination with the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the step of using a pre-trained embedding model to vectorize the obtained sentence to be detected to obtain a vector matrix for characterizing the sentence to be detected includes:

[0024] Based on the Embedding vocabulary built into the embedding model, the ID sequence of the sentence to be detected is obtained, and the Embedding vocabulary is traversed according to the ID sequence to obtain the Embedding vector corresponding to each ID in the ID sequence;

[0025] The Embedding vectors corresponding to each ID are concatenated to obtain a vector matrix for representing the sentence to be detected.

[0026] In combination with the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the step of using a pre-trained bidirectional long short-term memory network model to perform classification prediction processing on the vector matrix to obtain a predicted punctuation type corresponding to the punctuation to be detected based on the sentence to be detected includes:

[0027] Performing bidirectional feature extraction on the vector matrix to obtain context features of the sentence to be detected represented by the vector matrix;

[0028] According to the punctuation type preset in the bidirectional long short-term memory network model, the context feature is converted into a dimensional vector corresponding to the punctuation type;

[0029] The softmax function is used to normalize the dimension vector corresponding to the punctuation type into a probability value, and the punctuation type represented by the dimension corresponding to the maximum probability value is obtained as the predicted punctuation type corresponding to the punctuation to be detected.

[0030] In combination with the second possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the step of obtaining a list of incorrect punctuation corresponding to the target text content by comparing and merging the first detection result with the second detection result, and performing punctuation correction processing on the target text content according to the list of incorrect punctuation includes:

[0031] Determine whether the punctuation mark to be detected is within the detection range of the pre-configured error detection rule engine. If so, use the first detection result to merge the punctuation mark to be detected when comparing and merging the first detection result with the second detection result.

[0032] A second aspect of an embodiment of the present application provides a text punctuation correction device, the text punctuation correction device comprising:

[0033] An acquisition module is used to acquire target text content, where the target text content is text content that needs to be punctuation corrected and input by a user;

[0034] A detection module, used to perform punctuation error detection on the target text content using a pre-configured error detection rule engine and a pre-trained error detection model, respectively, to obtain a first detection result and a second detection result respectively;

[0035] The correction module is used to obtain a list of incorrect punctuation corresponding to the target text content by comparing and merging the first detection result with the second detection result, and perform punctuation correction processing on the target text content according to the list of incorrect punctuation.

[0036] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the electronic device, wherein the processor implements the steps of the text punctuation correction method provided in the first aspect when executing the computer program.

[0037] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the text punctuation correction method provided in the first aspect are implemented.

[0038] The text punctuation correction method, device, electronic device and storage medium provided in the embodiments of the present application have the following beneficial effects:

[0039] The present application obtains target text content, which is text content that needs to be corrected by punctuation input by the user; performs punctuation error detection on the target text content using a pre-configured error detection rule engine and a pre-trained error detection model, respectively, and obtains a first detection result and a second detection result accordingly; obtains a list of incorrect punctuation corresponding to the target text content by comparing and merging the first detection result with the second detection result, and performs punctuation correction on the target text content according to the incorrect punctuation list. The method performs error detection and correction on punctuation in the target text content by integrating the rule engine and the deep learning model, and has strong applicability and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A flowchart of a text punctuation correction method provided in the first embodiment of the present application;

[0042] Figure 2 A flow chart of a method for performing punctuation error detection using an error detection rule engine in a text punctuation correction method provided in an embodiment of the present application;

[0043] Figure 3 A flow chart of a method for detecting punctuation errors using an error detection model in a text punctuation correction method provided in an embodiment of the present application;

[0044] Figure 4 A flow chart of a method for preprocessing the sequence length of a detected sentence in a text punctuation correction method provided in an embodiment of the present application;

[0045] Figure 5 A schematic flow chart of a method for obtaining a vector matrix for representing a sentence to be detected in a text punctuation correction method provided in an embodiment of the present application;

[0046] Figure 6 A schematic flow chart of a method for performing classification prediction by an error detection model in a text punctuation correction method provided in an embodiment of the present application;

[0047] Figure 7 A structural block diagram of a text punctuation correction device provided in an embodiment of the present application;

[0048] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] Existing punctuation detection methods mainly use manual rules, but the punctuation scenarios that the rules can handle are very limited, and the effects vary depending on the usage scenarios. In most usage scenarios, voice transcription can be used to determine the punctuation position and type of text without punctuation based on model learning. Using traditional rules or predicting through audio, tone and other information can achieve punctuation detection tasks to a certain extent, but the overall accuracy is not high and the detection effect is not good. Especially in the scenario of writing meeting minutes, the correct use of punctuation is the part that recorders are most likely to overlook, such as the necessary writing requirements for minutes scenarios stipulated in the official document writing specifications (including the incorrect use of punctuation marks such as book title marks, half-width symbols, serial numbers, etc.) and conventional writing habits (including errors caused by mixed use of punctuation marks such as commas, semicolons, and periods). In the minutes scenario, firstly, the text information has no audio assistance, and the punctuation prediction method in the audio transcription based on tone cannot be applied; secondly, the minutes scenario itself has punctuation, and there is no need to judge the sentence segmentation of the text, only the type of punctuation usage needs to be corrected, which is different from the punctuation prediction task in audio transcription; thirdly, in the minutes scenario, the text has strict requirements on the use of punctuation in the overall framework, and some punctuation needs to be supplemented by strong rule judgment to correct the official document writing standards. The following provides a method for punctuation error detection that combines rule engines and deep learning models for scenarios such as meeting minutes that cannot be predicted by models only through rule engines or through audio, tone and other information. Taking the meeting minutes scenario as an example, the details are as follows:

[0051] See also Figure 1 , Figure 1 This is a flowchart of a text punctuation correction method provided in the first embodiment of the present application. The details are as follows:

[0052] Step S11: obtaining target text content, where the target text content is the text content input by the user and needs to be corrected by punctuation.

[0053] In this embodiment, the target text content is obtained from the text file input by the user, and the target text content can be all or part of the content in the text file. In this embodiment, in order to adapt to the format diversity of the text files uploaded by the user, when executing the text punctuation correction method, the system can also implement pre-processing such as content parsing of the text files uploaded by the user, so that the corresponding target text content can be obtained from the text files for text files of different formats. For example, in the meeting minutes scenario, when performing text punctuation correction processing, the user is first required to log in to the system to upload the meeting minutes file that needs to be reviewed, and the system receives the meeting minutes file uploaded by the user to obtain the target text content that needs to be corrected by punctuation from the meeting minutes file. In this embodiment, the meeting minutes file usually includes multiple minutes elements such as time, place, recorder, and text, and the target text content that needs to be corrected by punctuation is generally the content of the text part of the minutes. Therefore, when the system receives the meeting minutes file uploaded by the user, it can parse the meeting minutes file according to the minutes elements, break the meeting minutes file into multiple parts, and obtain the content of the minutes body as the target text content that needs punctuation correction.

[0054] Step S12: respectively using a pre-configured error detection rule engine and a pre-trained error detection model to perform punctuation error detection on the target text content, and correspondingly obtaining a first detection result and a second detection result.

[0055] In this embodiment, two methods for punctuation error detection of target text content are provided, wherein one is to use a pre-set engineered error detection rule engine for error detection. The other is to use an error detection model pre-trained based on a deep learning model for error detection. The detection result includes the error punctuation information detected in the target text content by the error detection rule engine. The second detection result includes the error punctuation information detected in the target text content by the error detection model. In this embodiment, the error detection rule engine can be used to perform error detection of general standard punctuation and error detection of common text punctuation collocation, detect whether the punctuation used in the target text content has general standard punctuation errors and / or common text punctuation collocation errors, and if so, judge that the punctuation used in the target text content is an incorrect punctuation. The error detection model can then perform punctuation prediction in combination with the contextual scene of the target text content, and synchronously compare whether the punctuation used in the target text content is consistent with the predicted punctuation. If not, judge that the punctuation used in the target text content is an incorrect punctuation.

[0056] Step S13: By comparing and merging the first detection result and the second detection result, a list of incorrect punctuation corresponding to the target text content is obtained, and punctuation correction processing is performed on the target text content according to the incorrect punctuation list.

[0057] In this embodiment, the first detection result is compared and merged with the second detection result, wherein, when the comparison and merging are performed, for the punctuation within the detection range of the error detection rule engine, the judgment result of the rule engine is preferably used. For example, for the same punctuation in the target text content, if the first detection result shows that the punctuation is an incorrect punctuation, and the second detection result does not show that the punctuation is an incorrect punctuation, the punctuation is still judged to be an incorrect punctuation. For punctuation that is not within the detection range of the error detection rule engine, the judgment result of the error detection model is used. After the first detection result is compared and merged with the second detection result, a list of incorrect punctuation corresponding to the target text content can be obtained, and according to the incorrect punctuation list, the incorrect punctuation in the target text content is corrected one by one. When performing punctuation correction processing, the correct punctuation can be determined based on the general standard punctuation usage rules and text punctuation matching usage rules in the error detection rule engine, or it can be determined by generating predicted punctuation through the error detection model.

[0058] From the above, it can be seen that the text punctuation correction method provided in this embodiment performs punctuation error detection on the target text content that needs to be corrected by punctuation using a pre-configured error detection rule engine and a pre-trained error detection model, respectively, to obtain a corresponding first detection result and a second detection result, and then compares and merges the first detection result and the second detection result to obtain an erroneous punctuation list corresponding to the target detection content, and performs punctuation correction processing according to the erroneous punctuation list. In this way, the error detection rule engine and the error detection model are integrated to obtain more accurate error detection results, effectively improving the correction rate of the correction process. Moreover, by integrating the error detection rule engine and the error detection model, the problem of usage scenario limitations that exist when a single rule engine is used for detection can also be compensated.

[0059] In some embodiments of this application, please refer to Figure 2 , Figure 2 A flowchart of a method for using an error detection rule engine to detect punctuation errors in a text punctuation correction method provided in an embodiment of the present application is provided. The details are as follows:

[0060] Step S21: obtaining punctuation to be detected in the target text content;

[0061] Step S22: traversing a first error type table based on a general standard punctuation configuration according to the punctuation to be detected, and determining whether the punctuation to be detected belongs to at least one error type in the first error type table; and / or traversing a second error type table based on a text punctuation collocation configuration according to the punctuation to be detected, and determining whether the punctuation to be detected belongs to at least one error type in the second error type table;

[0062] Step S23: If the punctuation mark to be detected belongs to at least one error type in the first error type table and / or the punctuation mark to be detected belongs to at least one error type in the second error type table, then the punctuation mark to be detected is marked as an erroneous punctuation mark in the first detection result.

[0063] In this embodiment, the first error type table may include but is not limited to the error types shown in Table 1. The first error type table is an error type table based on a general standard punctuation configuration.

[0064] Serial number Error Type 1 The punctuation used in the hierarchical sequence does not comply with the hierarchical punctuation rules: 1. (a) 1.1) ① Form 2 Use half-width punctuation in body text 3 Punctuation errors that need to appear in pairs 4 Use commas to separate multiple book titles or quotation marks. 5 The use of hyphens when indicating numerical values ​​and start and end years is not standardized 6 Same form of brackets 7 Use a period at the end of figure and table captions 8 The use of brackets when indicating the year of publication is not standard. 9 Use a comma to indicate a pause within a book title 10 Improper punctuation at the end of a sentence in brackets 11 Use punctuation after the attachment name

[0065] Table 1

[0066] In this embodiment, the punctuation to be detected is the punctuation that needs to be detected in the target text content. The punctuation to be detected is obtained in the target text content. One of the acquisition methods can be to start from the first punctuation of the target text content, and to obtain them one by one in order. After the detection of the currently acquired punctuation is completed, the next punctuation of the target text content is obtained for detection. When traversing the first error type table based on the general standard punctuation configuration according to the punctuation to be detected, the punctuation to be detected is compared with each error type in the first error type table to identify whether the punctuation to be detected meets the situation described by the error type. If it meets, it means that the punctuation to be detected belongs to this error type. If it does not meet, it means that the punctuation to be detected does not belong to this error type. When the punctuation to be detected is compared with all the error types in the first error type table and the comparison result is obtained, it can be determined whether the punctuation to be detected belongs to at least one error type in the first error type table according to the comparison result. If the punctuation to be detected belongs to at least one error type in the first error type table, the punctuation to be detected is marked as an error punctuation in the first detection result.

[0067] In this embodiment, the second error type table may include, but is not limited to, the error types shown in Table 2. The second error type table is an error type table based on text punctuation collocation configuration.

[0068]

[0069] Table 2

[0070] It can be understood that in this embodiment, when judging whether the punctuation mark to be detected belongs to at least one error type in the second error type table, its specific implementation method is basically the same as the above-mentioned implementation method of judging whether the punctuation mark to be detected belongs to at least one error type in the first error type table, and will not be repeated here.

[0071] In some embodiments of this application, please refer to Figure 3 , Figure 3 A flow chart of a method for detecting punctuation errors using an error detection model in a text punctuation correction method provided in an embodiment of the present application is provided. The details are as follows:

[0072] Step S31: breaking down the target text content into non-punctuation segments according to punctuation marks;

[0073] Step S32: obtaining a punctuation mark to be detected from the target text content, and assembling two punctuation-free sentence segments before and after the punctuation mark to be detected to obtain a sentence to be detected for detecting the punctuation mark to be detected;

[0074] Step S33: using a pre-trained embedding model to vectorize the obtained sentence to be detected, to obtain a vector matrix for representing the sentence to be detected;

[0075] Step S34: using a pre-trained bidirectional long short-term memory network model to perform classification prediction processing on the vector matrix, so as to obtain a predicted punctuation type corresponding to the punctuation to be detected based on the sentence to be detected;

[0076] Step S35: comparing the predicted punctuation type with the punctuation to be detected. If the punctuation to be detected is inconsistent with the predicted punctuation type, marking the punctuation to be detected as an erroneous punctuation in the second detection result.

[0077] In the present embodiment, the punctuation-free segment is the text information between two adjacent punctuation marks in the target text content. It is understandable that the first segment of each paragraph in the target text content is all the text information from the first word at the beginning of the paragraph to the last word before the first punctuation mark of the paragraph. In the present embodiment, the target text content includes at least one punctuation-free segment. After the target text content is disassembled into punctuation-free segments by punctuation, based on these punctuation-free segments, sentences to be detected corresponding to each punctuation mark to be detected in the target text content can be obtained, and the sentences to be detected are used to perform error detection on the punctuation marks to be detected in combination with the context scene. Specifically, the punctuation marks to be detected are obtained in the target text content, and generally all the punctuation marks in the target text content are the punctuation marks to be detected that need to be detected. For each punctuation mark to be detected, the sentences to be detected for detecting the punctuation marks to be detected are obtained by assembling the two punctuation-free segments before and after the punctuation marks to be detected, and the positions of the punctuation marks to be detected are marked with specific characters, such as the "#punc#" symbol. In this embodiment, when there is no punctuation-free segment before the punctuation to be detected or no punctuation-free segment after the punctuation to be detected in the target text content, a preset filler text is used to fill in the position where there is no punctuation-free segment before or after the punctuation to be detected. After obtaining the sentences to be detected corresponding to each punctuation to be detected, these sentences to be detected are input into a pre-trained embedding model, and the pre-trained embedding model is used to vectorize the obtained sentences to be detected, and a vector matrix corresponding to the sentences to be detected is generated. After the vector matrix is ​​generated, the vector matrix is ​​input into a pre-trained bidirectional long short-term memory network (BiLSTM) model, based on the six punctuation types of ".", "," ,": ","; "?", "!" divided during the training of the bidirectional long short-term memory network (BiLSTM) model, the pre-trained bidirectional long short-term memory network (BiLSTM) model is used to classify and predict the vector matrix, and the predicted punctuation type corresponding to the position of the "#punc#" symbol in the sentence to be detected is obtained. The predicted punctuation type is the punctuation type predicted by the error detection model for the punctuation to be detected. Furthermore, by comparing the predicted punctuation type with the punctuation to be detected, if the punctuation to be detected is inconsistent with the predicted punctuation type, it means that the error detection model detects that the punctuation to be detected is an erroneous punctuation. At this time, the punctuation to be detected is marked as an erroneous punctuation in the second detection result. In this embodiment, the error detection model obtained by deep learning training based on the Embedding model and the BiLSTM model can learn some explicit rules that cannot be exhaustively enumerated and some implicit rules that cannot be expressed when using punctuation.The error detection model can detect errors in the punctuation marks to be detected in the target text content based on these explicit rules and implicit rules. It has a wide range of usage scenarios and makes up for the problem that the rule engine cannot cover all usage scenarios.

[0078] In some embodiments of this application, please refer to Figure 4 , Figure 4 A flow chart of a method for preprocessing the sequence length of a detected sentence in a text punctuation correction method provided in an embodiment of the present application. Details are as follows:

[0079] Step S41: identifying the sequence length of the sentence to be detected;

[0080] Step S42: comparing the sequence length of the sentence to be detected with a preset sequence length threshold to obtain a comparison result;

[0081] Step S43: If the comparison result is that the sequence length of the sentence to be detected is less than the preset sequence length threshold, padding is performed on the sentence to be detected so that the sequence length of the sentence to be detected is consistent with the sequence length threshold preset by the model;

[0082] Step S44: If the comparison result is that the sequence length of the sentence to be detected is greater than the preset sequence length threshold, the sentence to be detected is truncated so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold.

[0083] In the present embodiment, a fixed sequence length threshold is preset for the input data in the error detection model. The data lengths of the sentences to be detected that are input into the error detection model vary. The error detection model can pre-process the sentences to be detected according to the preset sequence length threshold, so that the sequence lengths of the sentences to be detected are standardized, which is convenient for the subsequent vectorization processing of the sentences to be detected. Specifically, the sequence length of the sentence to be detected is first identified, and then the sequence length of the sentence to be detected is compared with the preset sequence length threshold, so that the size relationship between the two can be obtained. If the sequence length of the sentence to be detected is less than the preset sequence length threshold, the sentence to be detected is padded so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold. The text that fills the length of the sentence to be detected here is the padding text, and the character used for padding in this embodiment is set to be " <pad>", if the length of the sequence of the sentence to be detected differs by a few characters from the preset sequence length threshold, then add a few " at the end of the sentence to be detected <pad>" character. If the sequence length of the sentence to be detected is greater than the preset sequence length threshold, the sentence to be detected is truncated. For the sentence to be detected whose sequence length is greater than the preset sequence length threshold, a segment is truncated for each sequence length threshold until the last segment is less than or equal to the preset sequence length threshold. If the last segment is less than the preset sequence length threshold, the length is filled with padding text, so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold.

[0084] In some embodiments of this application, please refer to Figure 5 , Figure 5 A schematic flow chart of a method for obtaining a vector matrix for representing a sentence to be detected in a text punctuation correction method provided in an embodiment of the present application. The details are as follows:

[0085] Step S51: obtaining the ID sequence of the sentence to be detected based on the Embedding vocabulary built into the embedding model, and traversing the Embedding vocabulary according to the ID sequence to obtain the Embedding vector corresponding to each ID in the ID sequence;

[0086] Step S52: concatenate the Embedding vectors corresponding to each ID to obtain a vector matrix for representing the sentence to be detected.

[0087] In this embodiment, by inputting the sentence to be detected into the embedding model, the sentence to be detected can be character-vectorized based on the built-in Embedding vocabulary of the embedding model to obtain a vector matrix representing the sentence to be detected. Specifically, the embedding model maps the input sentence to be detected from the character space to the vector space by mapping, so as to realize the vector representation of the input sentence to be detected. In this embodiment, the embedding model has a built-in Embedding vocabulary created based on the corpus, and the Embedding vocabulary uses character granularity for word segmentation, that is, the basis of word segmentation is characters, and each character is a word. Since the characters cannot be calculated in the model, a corresponding ID is configured for each character in the Embedding vocabulary, and each ID corresponds to an Embedding vector. When the sentence to be detected is vectorized by characters, the ID corresponding to each character in the sentence to be detected is identified, and the ID sequence of the sentence to be detected is obtained. Then, according to the ID sequence, the Embedding vocabulary is traversed, and the Embedding vector corresponding to each character in the sentence to be detected can be retrieved based on the ID, and then the vector matrix for characterizing the sentence to be detected can be obtained by splicing the retrieved Embedding vectors, that is, the vector matrix is ​​the vectorized expression of the sentence to be detected. Specifically, in this embodiment, after the sentence to be detected is input into the embedding (Embedding) model, the ID corresponding to each character in the sentence to be detected is obtained by querying the Embedding vocabulary, and the ID sequence corresponding to the sentence to be detected is generated. Then the one-hot expression of the sentence to be detected is constructed using the ID sequence, wherein the one-hot expression is a vector with the same size as the Embedding vocabulary, and each dimension in the vector corresponds to a character in the Embedding vocabulary. According to the ID sequence of the sentence to be detected, the characters appearing in the sentence to be detected are mapped to the dimensions of the vector, that is, in the vector, the characters appearing in the sentence to be detected, and the corresponding dimension in the vector is set to 1, otherwise the dimension is set to 0. Thus, a sentence to be detected input into the embedding (Embedding) model can obtain a corresponding 01 vector expression. Then, the obtained 01 vector expression is multiplied by the Embedding matrix constructed based on the Embedding vocabulary to obtain the vector matrix representing the sentence to be detected. In addition, in this embodiment, the Embedding vocabulary is also configured with unknown characters " <unk>" and its corresponding ID and fill character " <pad>” and its corresponding ID. Among them, when character vectorizing the statement to be detected based on this Embedding vocabulary, unknown characters are used in scenarios where characters in the statement to be detected are not retrieved in the Embedding vocabulary, while padding text is used in scenarios where the sequence length of the statement to be detected is less than the fixed sequence length set by the Embedding model for the input text. For example, suppose the fixed sequence length set by the Embedding model is 8; the corpus is "I saw him walking slowly"; the Embedding vocabulary is "the character ID of 'I' is 0, the character ID of '看' is 1, the character ID of '见' is 2, the character ID of '他' is 3, the character ID of '慢' is 4, the character ID of '走' is 5, the character ID of '来' is 6, ' <pad>'The character ID is 7,' <unk>'The character ID is 8". If the sentence to be detected input into the embedding model is "I saw him eating", then the sentence to be detected is vectorized by character to obtain the ID sequence [0 1 2 3 8 8 7 7]. When the ID is reversed to text, it is "I saw him eating". <unk> <unk> <pad> <pad>", where the two characters "吃饭" in the sentence to be detected are not retrieved in the Embedding vocabulary, so they are regarded as unknown texts and are processed by <unk>The length of the sentence to be detected is 6, which is less than 8. Therefore, two characters are added to the end of the sentence to be detected to make its length reach the fixed sequence length set by the embedding model. The two added characters are replaced by <pad>Character replacement.

[0088] In some embodiments of this application, please refer to Figure 6 , Figure 6 A flow chart of a method for classifying and predicting an error detection model in a text punctuation correction method provided in an embodiment of the present application is provided. The details are as follows:

[0089] Step S61: performing bidirectional feature extraction on the vector matrix to obtain context features of the sentence to be detected represented by the vector matrix;

[0090] Step S62: converting the context feature into a dimensional vector corresponding to the punctuation type according to the punctuation type preset in the bidirectional long short-term memory network model;

[0091] Step S63: using a softmax function to normalize the dimension vector corresponding to the punctuation type into a probability value, and obtaining the punctuation type represented by the dimension corresponding to the maximum probability value as the predicted punctuation type corresponding to the punctuation to be detected.

[0092] In this embodiment, a bidirectional long short-term memory network (BiLSTM) model is used to perform the classification prediction task of punctuation. In the bidirectional long short-term memory network (BiLSTM) model, punctuation is divided into six types according to the punctuation type: ".", ",", ":", ";", "?", "!", and each punctuation type corresponds to a dimension. In this embodiment, a bidirectional feature extraction is performed on the vector matrix by a bidirectional long short-term memory network (BiLSTM) model to obtain the contextual features of the sentence to be detected represented by the vector matrix. Then, several linear layers of the bidirectional long short-term memory network (BiLSTM) model are transformed according to each punctuation type to obtain the dimensional vector corresponding to each punctuation type. Furthermore, the softmax function is used to normalize the dimensional vector obtained corresponding to each punctuation type into a probability value, and the size of the dimensional probability value corresponding to each punctuation type is compared. The larger the probability value, the more the punctuation type meets the requirements of the sentence to be detected represented by the vector matrix. Therefore, the punctuation type represented by the dimension corresponding to the maximum probability value is obtained as the predicted punctuation type corresponding to the punctuation to be detected. In this embodiment, the BiLSTM model usually defines the sentence before the punctuation in the sentence to be detected as the preceding text, and the sentence after the punctuation as the following text. The BiLSTM model obtains the prediction result of the information between the current time step (the preceding text) by performing positive calculations on the sentence to be detected from left to right; and obtains the prediction result of the current time step and the information after it (the following text) by performing reverse calculations on the sentence to be detected from right to left. For example, for the sentence to be detected "No choice but to let the flowers fall #punc#seemingly familiar, the swallows return", the BiLSTM model calculates once from left to right and then calculates once from right to left, which is equivalent to calculating the two sequences of "No choice but to let the flowers fall #punc#seemingly familiar, the swallows return" and "Returning swallows know each other #punc#going to fall flowers, he can't help but feel helpless", thus enabling the error detection model to predict the punctuation according to the context scenario through the two directions of positive and reverse order.

[0093] In some embodiments of the present application, when comparing and merging the first detection result and the second detection result, for punctuation within the detection range of the error detection rule engine, the judgment result of the rule engine is preferably used. In this embodiment, the pre-configured error detection rule engine can determine a fixed detection range based on its configured error type table, by judging whether the punctuation to be detected belongs to the detection range of the pre-configured error detection rule engine. If so, when comparing and merging the first detection result and the second detection result, the first detection result is used to merge the punctuation to be detected. In this way, the detection accuracy can be improved while satisfying the detection coverage, making up for the shortcomings of the error detection model in terms of accuracy.

[0094] See also Figure 7 , Figure 7 This is a structural block diagram of a text punctuation correction device provided in an embodiment of the present application. The various units included in the device in this embodiment are used to execute the various steps in the above method embodiment. For details, please refer to the relevant description in the above method embodiment. For ease of explanation, only the parts related to this embodiment are shown. Figure 7 As shown, the text punctuation correction device includes: an acquisition module 71, a detection module 72 and a correction module 73. Among them: the acquisition module 71 is used to obtain the target text content, and the target text content is the text content that needs to be corrected by punctuation input by the user. The detection module 72 is used to perform punctuation error detection on the target text content using a pre-configured error detection rule engine and a pre-trained error detection model, respectively, to obtain a first detection result and a second detection result accordingly. The correction module 73 is used to obtain a list of incorrect punctuation corresponding to the target text content by comparing and merging the first detection result with the second detection result, and perform punctuation correction on the target text content according to the incorrect punctuation list.

[0095] It should be understood that the above-mentioned text punctuation correction device corresponds one-to-one with the above-mentioned text punctuation correction method, and will not be described in detail here.

[0096] See also Figure 8 , Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 81, a memory 82, and a computer program 83 stored in the memory 82 and executable on the processor 81, such as a program of a text punctuation correction method. When the processor 81 executes the computer program 83, the steps in each embodiment of the above-mentioned text punctuation correction method are implemented. Alternatively, when the processor 81 executes the computer program 83, the functions of each module in the embodiment corresponding to the above-mentioned text punctuation correction device are implemented. Please refer to the relevant description in the embodiment for details, which will not be repeated here.

[0097] Exemplarily, the computer program 83 may be divided into one or more modules (units), which are stored in the memory 82 and executed by the processor 81 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 83 in the electronic device 8. For example, the computer program 83 may be divided into an acquisition module, a processing module, and an execution module, and the specific functions of each module are as described above.

[0098] The turntable device may include, but is not limited to, a processor 81 and a memory 82. Those skilled in the art will appreciate that Figure 8 It is only an example of the electronic device 8 and does not constitute a limitation of the electronic device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the turntable device may also include input and output devices, network access devices, buses, etc.

[0099] The processor 81 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0100] The memory 82 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. The memory 82 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 8. Further, the memory 82 may include both an internal storage unit and an external storage device of the electronic device 8. The memory 82 is used to store the computer program and other programs and data required by the turntable device. The memory 82 may also be used to temporarily store data that has been output or is to be output.

[0101] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0102] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented. In this embodiment, the computer-readable storage medium can be non-volatile or volatile.

[0103] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0104] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0105] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0106] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.< / pad> < / unk> < / pad> < / pad> < / unk> < / unk> < / unk> < / pad> < / pad> < / unk> < / pad> < / pad>

Claims

1. A method for correcting text punctuation, characterized in that: include: Acquire target text content, where the target text content is text content input by a user that needs to be punctuation corrected; Using a pre-configured error detection rule engine and a pre-trained error detection model to perform punctuation error detection on the target text content, respectively, to obtain a first detection result and a second detection result; By comparing and merging the first detection result with the second detection result, a list of incorrect punctuation corresponding to the target text content is obtained, and punctuation correction processing is performed on the target text content according to the incorrect punctuation list; The step of using a pre-trained error detection model to perform punctuation error detection on the target text content includes: Decomposing the target text content into non-punctuation segments according to punctuation marks; Obtaining a punctuation mark to be detected in the target text content, assembling two punctuation-free sentence segments before and after the punctuation mark to be detected, and obtaining a sentence to be detected for detecting the punctuation mark to be detected; Using a pre-trained embedding model to vectorize the obtained sentence to be detected, so as to obtain a vector matrix for representing the sentence to be detected; Using a pre-trained bidirectional long short-term memory network model to perform classification prediction processing on the vector matrix, so as to obtain a predicted punctuation type corresponding to the punctuation to be detected based on the sentence to be detected; Comparing the predicted punctuation type with the punctuation to be detected, if the punctuation to be detected is inconsistent with the predicted punctuation type, marking the punctuation to be detected as an erroneous punctuation in the second detection result; The step of using a pre-trained embedding model to vectorize the obtained sentence to be detected to obtain a vector matrix for representing the sentence to be detected includes: Based on the Embedding vocabulary built into the embedding model, the ID sequence of the sentence to be detected is obtained, and the Embedding vocabulary is traversed according to the ID sequence to obtain the Embedding vector corresponding to each ID in the ID sequence; The Embedding vectors corresponding to each ID are concatenated to obtain a vector matrix for representing the sentence to be detected.

2. The text punctuation correction method according to claim 1, characterized in that: The step of using a pre-configured error detection rule engine to perform punctuation error detection on the target text content comprises: Obtaining punctuation to be detected in the target text content; Traversing a first error type table based on a general standard punctuation configuration according to the punctuation to be detected, and determining whether the punctuation to be detected belongs to at least one error type in the first error type table; and / or traversing a second error type table based on a text punctuation collocation configuration according to the punctuation to be detected, and determining whether the punctuation to be detected belongs to at least one error type in the second error type table; If the punctuation mark to be detected belongs to at least one error type in the first error type table and / or the punctuation mark to be detected belongs to at least one error type in the second error type table, the punctuation mark to be detected is marked as an erroneous punctuation mark in the first detection result.

3. The text punctuation correction method according to claim 1, characterized in that: Before the step of using the pre-trained embedding model to vectorize the obtained sentence to be detected to obtain a vector matrix for representing the sentence to be detected, the method further includes: Identifying the sequence length of the sentence to be detected; Comparing the sequence length of the sentence to be detected with a preset sequence length threshold to obtain a comparison result; If the comparison result is that the sequence length of the sentence to be detected is less than the preset sequence length threshold, padding the sentence to be detected so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold; If the comparison result is that the sequence length of the sentence to be detected is greater than the preset sequence length threshold, the sentence to be detected is truncated so that the sequence length of the sentence to be detected is consistent with the preset sequence length threshold.

4. The text punctuation correction method according to claim 3, characterized in that: The step of using a pre-trained bidirectional long short-term memory network model to perform classification prediction processing on the vector matrix to obtain a predicted punctuation type corresponding to the punctuation to be detected based on the sentence to be detected includes: Performing bidirectional feature extraction on the vector matrix to obtain context features of the sentence to be detected represented by the vector matrix; According to the punctuation type preset in the bidirectional long short-term memory network model, the context feature is converted into a dimensional vector corresponding to the punctuation type; The softmax function is used to normalize the dimension vector corresponding to the punctuation type into a probability value, and the punctuation type represented by the dimension corresponding to the maximum probability value is obtained as the predicted punctuation type corresponding to the punctuation to be detected.

5. The text punctuation correction method according to claim 1, characterized in that: The step of obtaining a list of incorrect punctuation corresponding to the target text content by comparing and merging the first detection result with the second detection result, and performing punctuation correction processing on the target text content according to the list of incorrect punctuation includes: Determine whether the punctuation mark to be detected is within the detection range of the pre-configured error detection rule engine. If so, use the first detection result to merge the punctuation mark to be detected when comparing and merging the first detection result with the second detection result.

6. A text punctuation correction device, characterized in that: include: An acquisition module is used to acquire target text content, where the target text content is text content that needs to be punctuation corrected and input by a user; A detection module, used to perform punctuation error detection on the target text content using a pre-configured error detection rule engine and a pre-trained error detection model, respectively, to obtain a first detection result and a second detection result respectively; A correction module, configured to obtain a list of incorrect punctuation marks corresponding to the target text content by comparing and merging the first detection result with the second detection result, and perform punctuation correction processing on the target text content according to the list of incorrect punctuation marks; Among them, the detection module is also specifically used to decompose the target text content into punctuation-free segments according to punctuation; obtain the punctuation to be detected in the target text content, assemble the two punctuation-free segments before and after the punctuation to be detected, and obtain the sentence to be detected for detecting the punctuation to be detected; use a pre-trained embedding model to vectorize the obtained sentence to be detected to obtain a vector matrix for characterizing the sentence to be detected; use a pre-trained bidirectional long short-term memory network model to classify and predict the vector matrix to obtain the predicted punctuation type corresponding to the punctuation to be detected based on the sentence to be detected; compare the predicted punctuation type with the punctuation to be detected, and if the punctuation to be detected is inconsistent with the predicted punctuation type, mark the punctuation to be detected as an incorrect punctuation in the second detection result; The step of using a pre-trained embedding model to vectorize the obtained sentence to be detected to obtain a vector matrix for representing the sentence to be detected includes: obtaining an ID sequence of the sentence to be detected based on an Embedding vocabulary built into the embedding model, and traversing the Embedding vocabulary according to the ID sequence to obtain an Embedding vector corresponding to each ID in the ID sequence; and concatenating the Embedding vectors corresponding to each ID to obtain a vector matrix for representing the sentence to be detected.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images