Text correction method and device, computer device and readable storage medium

Abstract

The text correction method, device, computer equipment and readable storage medium provided by the application comprise: obtaining a character sequence corresponding to a text to be corrected, a confidence degree of each character in the character sequence and an image sequence composed of images of each character; obtaining semantic information features corresponding to the text to be corrected according to the character sequence and image information features according to the image sequence through a correction model; the correction model further comprises a first full connection layer, a transformer layer and a second full connection layer connected in sequence; predicting a candidate word set at each character position after fusing the semantic information features and the image information features; correcting a target character with a confidence degree less than a preset confidence threshold in the character sequence based on the candidate word set at the character position of the target character. The application fully considers semantic information and character shape information, determines candidate words according to fused feature information, more fully utilizes information and can help improve correction accuracy.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to a text correction method, apparatus, computer device, and readable storage medium. Background Technology

[0002] OCR (Optical Character Recognition) is the process of converting text in an image into a format that computers can directly process. This technology has wide applications in various business fields. For example, in the financial sector, OCR technology can be used to automatically extract statements and data from large amounts of financial reports, quickly obtaining information efficiently and at low cost, helping data users make investment decisions based on that information. However, OCR technology may produce some text recognition errors during the recognition process, making error correction of the recognition results essential.

[0003] Existing error correction methods primarily select correct characters from a pre-acquired confusion set to replace incorrect characters, thus completing the correction. This method requires extensive training on the corpus beforehand, which is time-consuming, labor-intensive, and cannot guarantee accuracy. Furthermore, correcting characters solely based on semantic fluency is a single metric with low efficiency. Therefore, current technologies suffer from low efficiency and low accuracy when correcting erroneous characters identified in image text. Summary of the Invention

[0004] One objective of this invention is to provide a text correction method, apparatus, computer device, and readable storage medium to improve the efficiency and accuracy of text correction. This invention can be achieved as follows:

[0005] In a first aspect, the present invention provides a text correction method, the method comprising: acquiring a text sequence corresponding to the text to be corrected, the confidence level of each character in the text sequence, and an image sequence composed of images of each character; obtaining semantic information features corresponding to the text to be corrected based on the text sequence, and obtaining image information features corresponding to the text to be corrected based on the image sequence; wherein the semantic information features and the image information features are obtained by a semantic information extractor and an image feature extractor of a pre-trained error correction model, respectively; the error correction model further comprises a first fully connected layer, a transformer layer, and a second fully connected layer connected in sequence; the first fully connected layer is used to process the semantic information features and the image information features. A linear transformation is performed on the information features to obtain the weight information of the semantic information features and the image information features respectively; the semantic information features and the image information features are fused, and based on the fused features, a set of candidate characters at each text position in the text to be corrected is predicted; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character; for target texts in the text sequence with a confidence level less than a preset confidence threshold, error correction is performed based on the set of candidate characters at the text position of the target text.

[0006] Secondly, the present invention provides a text correction device, comprising: an acquisition module, configured to: acquire a text sequence corresponding to the text to be corrected, the confidence level of each character in the text sequence, and an image sequence composed of images of each character; and a determination module, configured to: obtain semantic information features corresponding to the text to be corrected based on the text sequence, and obtain image information features corresponding to the text to be corrected based on the image sequence; wherein the semantic information features and the image information features are obtained by a semantic information extractor and an image feature extractor of a pre-trained correction model, respectively; the correction model further comprises a first fully connected layer, a transformer layer, and a second fully connected layer connected in sequence; the first fully connected layer is used to process the semantic information features and the image information features. The information features undergo a linear transformation to obtain the weight information of the semantic information features and the image information features respectively; the prediction module is used to fuse the semantic information features and the image information features, and predict the candidate character set at each text position of the text to be corrected based on the fused features; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character; the error correction module is used to correct the target text in the text sequence whose confidence is less than a preset confidence threshold based on the candidate character set at the text position of the target text.

[0007] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the method described in the first aspect.

[0008] Fourthly, the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0009] The present invention provides a text correction method, apparatus, computer device, and readable storage medium. The method includes: acquiring a text sequence corresponding to the text to be corrected, the confidence level of each character in the text sequence, and an image sequence composed of images of each character; obtaining semantic information features corresponding to the text to be corrected based on the text sequence, and obtaining image information features corresponding to the text to be corrected based on the image sequence; fusing the semantic information features and the image information features, and predicting a set of candidate characters at each character position in the text to be corrected based on the fused features; and correcting target characters in the text sequence whose confidence level is less than a preset confidence threshold based on the set of candidate characters at the character position of the target character. The entire process fully considers semantic and glyph information, determines candidate characters based on the fused feature information, makes fuller use of information, and can help improve the accuracy of error correction. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0012] Figure 2 This is a schematic diagram of the structure of the error correction model provided in an embodiment of the present invention;

[0013] Figure 3 A schematic flowchart illustrating the text correction method provided in an embodiment of the present invention;

[0014] Figure 4 A schematic flowchart of step 304 provided in an embodiment of the present invention;

[0015] Figure 5 A schematic flowchart of step 306 provided in an embodiment of the present invention;

[0016] Figure 6 A functional block diagram of the text correction device provided in the embodiments of the present invention;

[0017] Figure 7 A structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0022] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0023] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0024] Please see Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application. The scenario includes: a user 10, a terminal device 11, and a server 12. The terminal device 11 may have various client applications installed. After establishing a communication connection between the client of the terminal device 11 and the server 12 through a communication network, the client of the terminal device 11 can send the content to be identified and corrected to the server 12. The server 12 then identifies and corrects the content, obtains the correction result, and sends the correction result back to the client of the terminal device 11.

[0025] Among them, the terminal device 11 may be, but is not limited to, a scanner, a digital camera, various personal computers with image information acquisition functions, laptops, smartphones, tablets, smart wearable devices and other computer equipment.

[0026] Server 12 can be implemented using a standalone server, a server cluster composed of multiple servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Terminal device 11 and server 12 can be connected directly or indirectly via wired or wireless communication; this application does not impose any restrictions on this connection.

[0027] Terminal device 11 and server 12 can communicate with each other through a communication network. The communication network can be a wireless fidelity (Wi-Fi) hotspot network, Bluetooth (BT) network, or near field communication (NFC) network, or it can be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future public land mobile network (PLMN), or the Internet.

[0028] See also Figure 1 Currently, the process of text correction is as follows: An image to be recognized is input, and then OCR technology is used to recognize the image, obtaining the recognition result. The text in the recognition result is then used as the text to be corrected. For this text, there are currently three main error correction methods. Research has found that all of these methods have shortcomings:

[0029] The first method: error correction based on confusion sets.

[0030] The system detects which characters are incorrect based on custom validation rules or statistical probability calculations. Then, it selects the correct characters from a confusion set to replace the incorrect ones, thus completing the error correction. The confusion set is based on an accumulated business corpus or is pre-generated according to certain rules (such as similar-looking or similar-sounding characters). The process of selecting the correct characters is the error correction process, which may include several sub-methods. The main sub-methods include: calculating the similarity between characters in the confusion set and the incorrect characters; calculating the similarity or edit distance between the short text containing the incorrect characters and the new short text obtained by replacing the incorrect characters with characters from the corpus; and calculating similarity scores according to custom rules.

[0031] However, in OCR recognition scenarios, the confusion set may not cover all candidate characters, and the error correction result tends to predict characters in the confusion set or only characters in it, thus affecting the error correction capability. Moreover, it cannot cover all similar relationships, nor can it distinguish the differences among similar characters. Therefore, when correcting OCR results, it may not be able to fully utilize the similarities and differences in pronunciation, form, and meaning of Chinese characters, thus affecting the error correction capability.

[0032] The second method is model-based error correction.

[0033] The model obtains candidate characters and their corresponding confidence scores from the OCR output, combines the candidate characters based on the confidence scores, and ultimately uses the model as a tool to check the fluency or semantics of the character combinations and select the appropriate character combination; or the model is a preset probability statistical model that selects the character combination with the highest probability.

[0034] However, from the perspective of model-based error correction, if the model is only used as a tool to ultimately verify fluency, then only the semantic and sequential properties of Chinese characters are considered. Correspondingly, when using a language model, if only the candidate characters are generated by it, only semantic information is considered, while the features of the Chinese characters themselves are not considered, such as the glyph structure. The ignored glyph information may also affect the error correction capability.

[0035] The third method is an error correction method that combines confusion sets and models.

[0036] Error detection is performed using a model. During error correction, a preliminary selection of candidate characters is obtained using a confusion set, and then the model is used for further selection. The optimal result is selected by judging semantics or sentence fluency. Alternatively, the model can learn the feature information of characters in the confusion set and then correct the input text.

[0037] While this combined approach can improve the error correction effect, it still only corrects characters based on semantic fluency during the error correction process. The indicator is singular and the accuracy cannot meet the user's needs.

[0038] Based on the above summary and analysis of the limitations of existing technologies, this application provides a text correction method that avoids using the aforementioned confusion set during the correction process, thus minimizing its shortcomings. Furthermore, considering that some existing methods do not utilize glyph information, this application adds the use of glyph information, providing more feature information to the correction model. This aligns text image information features with text semantic information features, resulting in more efficient information utilization and improved correction accuracy.

[0039] Before introducing the text correction method provided in the embodiments of the present invention, the error correction model involved in the embodiments of the present invention will be introduced first.

[0040] Please see Figure 2 , Figure 2 This is a schematic diagram of the error correction model provided in an embodiment of the present invention. The error correction model may include: a semantic information extractor, an image feature extractor, a first fully connected layer, a transformer layer, and a second fully connected layer. The connection order of each model structure is as follows: Figure 2 As shown.

[0041] The semantic information extractor receives the input text sequence and outputs semantic information features. It can be trained using corpora relevant to the business scenario, and existing model training methods can be used, which will not be elaborated upon here. The image feature extractor receives the input image sequence and outputs image information features.

[0042] As an alternative implementation, the semantic information extractor may be, but is not limited to, a BERT model, and the image feature extractor may be, but is not limited to, a stacked n-layer ResNet residual network.

[0043] The first fully connected layer is used to perform a linear transformation on the semantic information features extracted by the semantic information extractor and the image information features extracted by the image feature extractor, and input the result of the linear transformation into the sigmoid function to obtain the weight information of the semantic information features and the image information features respectively.

[0044] The transformer layer is used to understand and fuse semantic and image information features at the sentence level to obtain fused features of semantic and image information. The number of transformer layers can be set according to actual needs; for example, a transformer structure with 3 to 6 layers can be set.

[0045] The fused features obtained from the transformer layer are processed by the second fully connected layer. Then, the processed fused features are normalized by layer. The normalized fused features are then input into the softmax function to solve for the probability distribution of each text position belonging to a certain character.

[0046] It is understandable that, from a model structure perspective, the error correction model in this application is completely different from existing text error correction models. From a functional perspective, the various components of the error correction model in this application have different functions. The first fully connected layer is used to perform linear transformation on the semantic information features and the image information features; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character. The various components of the above error correction model and their corresponding technical features support each other functionally and have an interactive relationship, thereby achieving the effect of accurate error correction.

[0047] As an optional implementation, the error correction model in this application embodiment may not integrate the semantic information extractor and the image feature extractor. The error correction model, the semantic information extractor, and the image feature extractor are implemented as independent functional modules in a collaborative manner to achieve the text error correction method.

[0048] The following will refer to the relevant accompanying drawings and... Figure 2 The error correction model provided in this application will be described in detail for the text error correction method provided in the embodiments of this application.

[0049] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating the text correction method provided in an embodiment of the present invention. The execution entity of this method can be... Figure 1 Server 12 in the middle, the method includes:

[0050] S300: Obtain the text sequence corresponding to the text to be corrected, the confidence level of each text in the text sequence, and the image sequence composed of the images of each text.

[0051] S302. Based on the text sequence, obtain the semantic information features corresponding to the text to be corrected, and based on the image sequence, obtain the image information features corresponding to the text to be corrected; wherein, the semantic information features and the image information features are obtained through the semantic information extractor and image feature extractor of the pre-trained error correction model, respectively; the error correction model further includes a first fully connected layer, a transformer layer, and a second fully connected layer connected in sequence; the first fully connected layer is used to perform a linear transformation on the semantic information features and the image information features to obtain the weight information of the semantic information features and the image information features respectively;

[0052] S304. The semantic information features and image information features are fused together, and based on the fused features, the candidate character set at each text position in the text to be corrected is predicted; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character;

[0053] S306. For target texts in a text sequence with a confidence level lower than a preset confidence threshold, perform error correction based on the set of candidate characters at the text position of the target text.

[0054] In the above text correction method, the text sequence corresponding to the text to be corrected and the image sequence corresponding to the text sequence are first obtained. Semantic information features and image information features are obtained through the semantic information extractor and image feature extractor of the correction model. Then, the first fully connected layer of the correction model is used to perform linear transformation on the semantic information features and image information features. Then, the transformer layer is used to understand and fuse the semantic information features and image information features at the sentence level. Finally, the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character. It can be seen that the various components of the above correction model and their corresponding technical features support each other and have an interactive relationship in terms of function. By fusing semantic information features and image information features, and then predicting the candidate character set at each text position based on the fused features, the correction is performed on the target text based on the candidate character set at the text position of the target text. The whole process fully considers semantic information and glyph information, and determines candidate characters based on the fused feature information. Information utilization is more efficient, which can help improve the accuracy of error correction.

[0055] The following is a detailed description of steps S300 to S306.

[0056] In step S300, obtain the character sequence corresponding to the text to be corrected, the confidence of each character in the character sequence, and the image sequence composed of the images of each character.

[0057] The above text to be corrected is the text recognized from the image to be recognized based on the existing character recognition technology. The image to be recognized can be a picture of a certain page in a portable document format (PDF) file, or it can also be a flyer, a poster, etc. The present application does not limit this.

[0058] As an optional implementation manner, the server 12 can first obtain the image to be recognized, and then use the OCR technology to recognize the image to be recognized, and obtain the OCR recognition text as the text to be corrected in the embodiments of the present application. For example, convert the characters on the image into an encodable text format. For the specific implementation of the prior art, reference can be made here, and no further description will be given here.

[0059] Therefore, for the above step S300, an embodiment of the present invention gives a possible implementation manner, that is, step S300 may include the following steps:

[0060] a1: Recognize the image of the text to be recognized based on the optical character recognition model, and output the character sequence, the confidence of each character in the character sequence, and the coordinate values of each character;

[0061] a2: Based on the coordinate values of each character, cut the image of the text to be recognized to obtain an image sequence.

[0062] For example, the text content in the image of the text to be recognized is "merge reserves", and the recognition result by the optical character recognition model is: he, bing, chu, bei. The confidence value corresponding to each character is [l1, l2, l3, l4], and the position of each character in the image coordinate system where the image to be recognized is located is [[x1, y], [x2, y], [x3, y].[x4, y]].

[0063] The confidence can be used to determine which characters need to go through the subsequent error correction process.

[0064] The coordinates can be used to indicate accurately cutting out the image corresponding to each character from the image of the text to be recognized. For example, the vertical projection can be calculated for the image of the text to be recognized, the combined projection is calculated to obtain the segmentation box, and the image is cropped according to the segmentation box. The purpose is to cut the character sequence in the original image into multiple images containing only single characters in sequence, and each single-character image corresponds to the parsed single character.

[0065] It should be noted that the above-mentioned text image to be recognized is only the image corresponding to one line of text in the image to be recognized. Specifically, it can be cropped from the image to be recognized, and the image corresponding to each line of text can be used as the text image to be recognized. In this way, the problem of text order alignment that occurs when performing text recognition on the entire image to be recognized can be avoided.

[0066] It should also be noted that, in order to meet the data format requirements of the subsequent error correction model, after obtaining the text sequence, the sequence length is controlled to be less than or equal to a preset length threshold, such as 125. If the obtained sequence length is greater than the preset length threshold, the original text sequence is split to ensure that the length of each sub-text sequence is less than the preset length threshold.

[0067] After obtaining the text sequence and image sequence, step S302 can be executed.

[0068] In step S302, the semantic information features corresponding to the text to be corrected are obtained according to the text sequence, and the image information features corresponding to the text to be corrected are obtained according to the image sequence.

[0069] In this embodiment of the application, semantic information features are obtained through... Figure 2 The semantic information extractor of the error correction model shown is obtained.

[0070] As an optional implementation, the semantic information extractor may be, but is not limited to, a pre-trained BERT model. This BERT model can be trained on corpora from different business scenarios, resulting in enhanced semantic understanding in the corresponding business scenarios, thereby achieving text correction effects in different business scenarios. Business scenarios may include, but are not limited to, finance, social networking, live streaming, games, etc., and this application embodiment does not limit them.

[0071] In order to accurately extract semantic information, the semantic information extractor in the embodiments of this application can be a multi-layer transformer structure. As a basic architecture in the field of natural language understanding, this structure can output the semantic representation of the text in the last layer of the entire structure. The specific details of the transformer structure will not be described in detail here.

[0072] In this embodiment, semantic information features are represented by a matrix, denoted as H. s =(h1) s h2 s ,…h N s ), H s h1 is the symbolic representation of the semantic information feature matrix. s h2 s ,…h Ns Let N be the semantic information representation of each character, where N is the length of the pre-encoded character sequence.

[0073] Therefore, this application also provides an implementation method for extracting semantic information features, as shown in b1 to b2 below:

[0074] b1: Pre-encode the text sequence;

[0075] b2 inputs the pre-encoded text sequence into the semantic information extractor of the pre-trained error correction model to obtain semantic information features.

[0076] It is understandable that the purpose of precoding text sequences is to convert them into a data format that a semantic information extractor can process. Taking the BERT model as an example, before inputting a text sequence into the BERT model, the text sequence needs to be converted, that is, precoded into a string of data. The length of the data, which is N mentioned above, can be set according to actual needs and is not limited here.

[0077] It should be noted that N can be selected based on the data format requirements of the semantic information extractor in this application, and no restrictions are imposed here.

[0078] While extracting semantic information features, we can also extract image information features corresponding to each character based on the image sequence. The image information features mentioned here can also be understood as the glyph structure features of the characters. It is conceivable that combining semantic information and glyph information for text correction can improve the accuracy of the correction results.

[0079] In this embodiment of the application, image information features can be obtained through Figure 2 The image feature extractor shown extracts the features.

[0080] As an alternative implementation, the image feature extractor may be, but is not limited to, a stacked n-layer ResNet residual network.

[0081] This application also provides an implementation method for extracting image information features, as shown in c1 to c2 below:

[0082] c1: Convert each image in the image sequence into a matrix of a preset size;

[0083] c2: Concatenate the obtained matrices according to the order of the images in the image sequence, and input the concatenated matrix into the image feature extractor of the pre-trained error correction model to obtain image information features.

[0084] To improve processing speed, each image in the image sequence can be converted into a matrix of a preset size in grayscale mode. It's important to note that this preset size must match the number of layers in the image feature extractor. That is, if the image feature extractor has n layers and each image is converted into an m*m*1 matrix, then n and m must satisfy the following relationship: 2 n =m, which means that if the ResNet network has 4 layers, then each image is converted into a 16*16*1 matrix after loading.

[0085] The matrix obtained after the above transformation is input sequentially into the image feature extractor. The image information features extracted by the image feature extractor can be denoted as H. g =(h1) g h2 g ,…h N g ), H g h1 is a symbolic representation of image (graphic) information features. g h2 g ,…h N g Representing the image information corresponding to each image.

[0086] After obtaining the semantic information features and image information features corresponding to the text to be corrected, step S304 can be executed.

[0087] In step S304, semantic information features and image information features are fused, and based on the fused features, a set of candidate characters at each text position in the text to be corrected is predicted.

[0088] In this embodiment, the obtained semantic information feature H s and image information features H g By concatenating the features along the feature dimension, it's equivalent to expanding the features of semantic information and glyphs together, resulting in matrix H. a =(h1) s,g h2 s,g ,…h N s ,g ), where H a It can be seen as a representation of the features after fusion.

[0089] The aforementioned "fusion" is equivalent to aggregating semantic and image information features at each location. For example, H s H is a matrix of dimension (2, 128, 768). g The matrix is ​​of dimension (2, 128, 768), where 2 represents the number of sentences and 128 represents the number of characters in each sentence. H aThe matrix is ​​the result of concatenating the two in the last dimension, which is (2, 128, 768*2).

[0090] It should be noted that in the above example, H s The dimension is (2, 128, 768), where 2 represents the number of sentences. During the training phase, to improve training efficiency, the number of sentences is equivalent to the number of training statements in a training batch. However, during the model application phase, H... s The number of sentences in a dimension is usually set to 1, representing a text to be corrected.

[0091] After obtaining the fused features, the candidate character set at each character position can be predicted.

[0092] In this embodiment, candidate characters are derived from a pre-established dictionary. Based on the fused features, this embodiment can predict the probability that a candidate character in the dictionary belongs to each character position in the text to be corrected, and then determine the candidate character set based on the probability. The candidate character set can be used to correct characters at certain positions that require correction.

[0093] Therefore, this application also provides an implementation of step S304, please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic flowchart of step S304 provided in an embodiment of this application. Step S304 may include the following steps:

[0094] S304-1, based on semantic information weights and image information weights, performs a weighted summation of semantic information features and image information features to obtain the fused features.

[0095] As an optional implementation, the semantic information weights and image information weights can be preset according to actual needs, and the relationship between the semantic information weights and image information weights is not limited.

[0096] As a preferred implementation method, the following d1 to d2 can be determined based on the implementation method provided in this application:

[0097] d1: Perform feature concatenation on semantic information features and image information features, and construct a weight matrix based on the concatenated features and preset weight parameters.

[0098] In this embodiment of the application, semantic information features and image information features are concatenated, that is, the above-mentioned H... s and H g Perform feature concatenation to obtain H a The obtained H a Input to Figure 2The first fully connected layer shown performs a linear transformation to fuse semantic and image information through a weighted summation. The result of the transformation is: W·H a +b, where W is the initial weight matrix, the parameters of which can be learned, and b is a constant vector.

[0099] d2: Solve the weight matrix using the activation function to output the semantic information weights and image information weights.

[0100] The result of the linear transformation is solved using an activation function to obtain the final weight matrix G = sigmoid(W·H). a +b), sigmoid is the activation function, and then G is changed to 2 in the dimension of the output features, resulting in two weight matrix components: w s =G[:,:,0] and w g =G[:,:,1], where w s For semantic information weights, w g The weights are for image information.

[0101] Using the obtained w s and w g semantic information features H s and image information features H g We obtain w by performing a weighted summation. s ·H s +w g ·H g That is, H, i.e., H = w s ·H s +w g ·H g .

[0102] The feature H obtained in the above steps is used as Figure 2 The feature representation is performed on the sentence-level understanding and fusion of the transformer structure.

[0103] As an optional implementation, the number of transformer layers can be set to 3-6 layers, and there is no limitation here.

[0104] S304-2, the fused features are input into the normalized exponential function to predict the probability that each candidate character in the pre-built dictionary belongs to each character position.

[0105] In this embodiment, the dictionary is preset by the semantic extractor itself. Before inputting H into the normalized exponential function, in order to meet the data format processed by the normalized exponential function, H can be normalized first, that is, H is processed sequentially through... Figure 2The second fully connected layer and the normalization layer are shown. Then, the normalized H is input into the normalization exponential function, which is the softmax function. The result of the normalization exponential function is a probability matrix. Each vector in the probability matrix is ​​the probability vector corresponding to each text position. Each vector position in a probability vector represents the index of a candidate character in the dictionary. The probability at the vector position is the probability of the candidate character corresponding to that index. Thus, the probability distribution of each text position belonging to a certain text is obtained.

[0106] S304-3, For each character position, multiple candidate characters are determined from the pre-established dictionary in descending order of probability value, forming a candidate character set.

[0107] For each character position, select the top K candidate characters corresponding to its probability values ​​from the probability vector in descending order of probability value to form a candidate character combination.

[0108] For example, if K is 4, the probability vector is [p1, p2, p3, p4, ... p n ], where p1>p2>p3>p4>…>p n Then, take the candidate characters corresponding to the first p1, p2, p3, and p4 respectively to form a candidate character set.

[0109] Once the candidate character set for each character position is obtained, the error correction process in this embodiment of the invention can be executed. See step S306.

[0110] In step S306, for target texts in the text sequence with a confidence level less than a preset confidence threshold, error correction is performed based on the set of candidate characters at the text position of the target text.

[0111] In this embodiment, the confidence score for each character can be obtained simultaneously with the text sequence. This confidence score represents the confidence level of correct character recognition. Therefore, a confidence score is determined for each character. A threshold is pre-set. If the confidence score is higher than the threshold, the parsing result is considered to match the content corresponding to the original image, and the parsing is considered correct, requiring no error correction. For characters with a confidence score lower than the pre-set threshold, error correction is required. Therefore, after obtaining the target characters with a confidence score lower than the pre-set threshold, error correction can be performed based on the candidate character set corresponding to the target characters.

[0112] This invention provides an implementation method for error correction based on a candidate character set. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic flowchart illustrating step S306 provided in an embodiment of this application. Step S306 may include:

[0113] S306-1, The candidate character with the highest probability in the candidate character set is determined as the character to be confirmed;

[0114] S306-2 If the character to be confirmed meets the preset conditions, then the target character is determined to be the correct character.

[0115] The preset conditions are: the character to be confirmed is the same as the target character, and the maximum probability is greater than the preset probability threshold.

[0116] The reason for setting the above preset conditions is that errors may occur during the target text recognition process, that is, correct text may be identified as incorrect text. The probability distribution of a certain text position predicted by the feature that integrates image information and semantic information can be used as a reference for the correct text. Therefore, if the character to be confirmed is the same as the target text, but the probability is greater than the preset probability threshold, it means that the target text identified in the early stage and the character to be confirmed predicted based on the target text are both correct. If this preset condition is not met, it means that the target text in the early stage and the character to be confirmed predicted based on the target text may be incorrect.

[0117] If the character to be confirmed meets the above preset conditions, the target character at the character position is considered to be correct and no error correction is required; otherwise, step S306-3 is executed.

[0118] S306-3 If the character to be confirmed does not meet the preset conditions, the target character is replaced with the character to be confirmed, or, based on the stroke similarity between the target character and each candidate character in the candidate character set and the probability of each candidate character, the target candidate character is determined and the target character is replaced with the target candidate character.

[0119] If the character to be confirmed does not meet the above preset conditions, it indicates that the target character is incorrect. In this case, the target character can be corrected in any of the following ways.

[0120] As an optional implementation, the target text can be replaced with the text to be confirmed, thus completing the error correction of the target text.

[0121] As another optional implementation method, the target candidate character is determined based on the stroke similarity between the target character and each candidate character in the candidate character set, as well as the probability of each candidate character. The target character is then replaced with the target candidate character, thus completing the error correction of the target character.

[0122] Specifically, the target text and the candidate characters in the candidate character set are decomposed into stroke sequences arranged in stroke order. Then, the similarity between the candidate characters and the target text is calculated. The candidate character with the maximum similarity and probability is selected as the target candidate character. The weights of similarity and probability may be selected based on multiple experimental experiences or statistical results.

[0123] It should be noted that this application embodiment only corrects Chinese text. If the value at a certain position of the text to be corrected is a number, letter or symbol other than a Chinese character, it will not be corrected and the value at the corresponding position will be used directly. Similarly, if there are candidate words with numbers, letters or symbols in the candidate word set, these candidate words can also be excluded from the candidate word set and selected from the remaining candidate word set.

[0124] It should also be noted that the model itself may introduce some miscorrections for words such as place names and organizations. To further improve the model's accuracy, if a usable dictionary of place names, organization names, and proper nouns is available, a Neural Error Correction (NER) model can be introduced. First, the OCR results are fed into the NER to identify possible place names, organization names, and proper nouns. Based on the aforementioned error correction model, for erroneous locations, the entities identified by the NER, including those at the erroneous locations, can be compared with the names in the dictionary. Words in the dictionary are used as candidate words, and finally, appropriate replacement words are selected based on metrics such as edit distance or similarity.

[0125] Based on the above embodiments, this application also provides a text correction device, the structure of which is as follows: Figure 6 As shown, Figure 6 The present invention provides a functional block diagram of a text correction device 400, which may include: an acquisition module 401, a determination module 402, a prediction module 403, and a correction module 404.

[0126] The acquisition module 401 is used to: acquire the text sequence corresponding to the text to be corrected, the confidence of each character in the text sequence, and the image sequence composed of the images of each character;

[0127] The determining module 402 is used to: obtain semantic information features corresponding to the text to be corrected based on the text sequence, and obtain image information features corresponding to the text to be corrected based on the image sequence; wherein the semantic information features and the image information features are obtained by the semantic information extractor and the image feature extractor of a pre-trained error correction model, respectively; the error correction model further includes a first fully connected layer, a transformer layer and a second fully connected layer connected in sequence; the first fully connected layer is used to perform a linear transformation on the semantic information features and the image information features to obtain the weight information of the semantic information features and the image information features respectively;

[0128] The prediction module 403 is used to fuse semantic information features and image information features, and based on the fused features, predict the candidate character set at each text position in the text to be corrected; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character;

[0129] The error correction module 404 is used to correct errors for target texts in a text sequence whose confidence level is less than a preset confidence threshold, based on the set of candidate characters at the text position of the target text.

[0130] It is understandable that the acquisition module 401, the determination module 402, the prediction module 403, and the error correction module 404 can be executed collaboratively. Figure 3 Each step in the process is used to achieve the corresponding technical effect.

[0131] In an optional implementation, the acquisition module 401 is specifically used to: recognize the text image to be recognized based on an optical character recognition model, and output a text sequence, the confidence level of each character in the text sequence, and the coordinate value of each character; and based on the coordinate value of each character, segment the text image to be recognized to obtain an image sequence.

[0132] In an optional implementation, the determining module 402 is specifically used for: pre-encoding the text sequence; inputting the pre-encoded text sequence into the semantic information extractor of the pre-trained error correction model to obtain semantic information features; converting each image in the image sequence into a matrix of a preset size; concatenating the obtained matrices according to the order of the images in the image sequence, and inputting the concatenated matrix into the image feature extractor of the pre-trained error correction model to obtain image information features.

[0133] In an optional implementation, the prediction module 403 is specifically used to: perform a weighted summation of semantic information features and image information features based on semantic information weights and image information weights to obtain fused features; input the fused features into a normalized exponential function to predict the probability that each candidate character in the pre-established dictionary belongs to each character position; and for each character position, determine multiple candidate characters from the pre-established dictionary in descending order of probability values ​​to form a candidate character set.

[0134] In an optional implementation, the prediction module 403 is specifically used to: perform feature concatenation on semantic information features and image information features, and construct a weight matrix based on the concatenated features and preset weight parameters; solve the weight matrix using an activation function, and output the semantic information weights and image information weights.

[0135] In an optional implementation, the error correction module 404 is specifically used to: determine the candidate character corresponding to the maximum probability in the candidate character set as the character to be confirmed; if the character to be confirmed meets a preset condition, then determine the target text as the correct text; the preset condition is: the character to be confirmed is the same as the target text, and the maximum probability is greater than a preset probability threshold; if the character to be confirmed does not meet the preset condition, then replace the target text with the character to be confirmed; or, based on the stroke similarity between the target text and each candidate character in the candidate character set and the probability of each candidate character, determine the target candidate character and replace the target text with the target candidate character.

[0136] It should be noted that the module division in the above embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical entities, or have two or more units integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] Based on the above embodiments, this application also provides a schematic diagram of a computer device used to implement the text correction method in the above embodiments. (See also...) Figure 7 As shown, the computer device 500 includes a memory 501, a processor 502, a communication interface 503, and a bus 504. The memory 501, processor 502, and communication interface 503 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0139] Optionally, bus 504 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0140] In this embodiment, the processor 502 may be a general-purpose processor, digital signal processor, application-specific integrated circuit, field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment can be directly implemented by the hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules may reside in the memory 501, and the processor 502 reads the program instructions from the memory 501 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0141] In this embodiment, the memory 501 can be non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as RAM. The memory can also be any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. The memory in this embodiment can also be a circuit or any other device capable of implementing storage functions, used to store instructions and / or data.

[0142] The memory 501 can be used to store software programs and modules, such as the instructions / modules of the text correction device 400 provided in this embodiment of the invention. These can be stored in the memory 501 in the form of software or firmware, or embedded in the operating system (OS) of the computer device 500. The processor 502 executes various functional applications and data processing by executing the software programs and modules stored in the memory 501. The communication interface 503 can be used to communicate with other node devices for signaling or data.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] Understandable. Figure 7 The structure shown is for illustrative purposes only; the computer device 500 may also include more than [other components]. Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0145] Based on the above embodiments, this application also provides a storage medium storing a computer program, which, when executed by a computer, causes the computer to perform the text correction method provided in the above embodiments.

[0146] Based on the above embodiments, this application also provides a computer program that, when run on a computer, causes the computer to execute the text correction method provided in the above embodiments.

[0147] Based on the above embodiments, this application also provides a chip for reading a computer program stored in a memory and executing the text correction method provided in the above embodiments.

[0148] This application also provides a computer program product, including instructions that, when run on a computer, cause the computer to execute the text correction method provided in the above embodiments.

[0149] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A text error correction method, characterized in that, The method includes: Obtain the text sequence corresponding to the text to be corrected, the confidence level of each character in the text sequence, and the image sequence composed of the images of each character; Based on the text sequence, semantic information features corresponding to the text to be corrected are obtained, and based on the image sequence, image information features corresponding to the text to be corrected are obtained. The semantic information features and the image information features are obtained through the semantic information extractor and image feature extractor of the pre-trained error correction model, respectively. The error correction model further includes a first fully connected layer, a transformer layer, and a second fully connected layer connected in sequence. The first fully connected layer is used to perform a linear transformation on the semantic information features and the image information features to obtain the weight information of the semantic information features and the image information features, respectively. The semantic information features and the image information features are fused together, and based on the fused features, a set of candidate characters at each text position in the text to be corrected is predicted; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character; For target characters in the text sequence with a confidence level less than a preset confidence threshold, error correction is performed based on a set of candidate characters at the position of the target character. This includes: identifying the candidate character with the highest probability in the candidate character set as the character to be confirmed; if the character to be confirmed does not meet a preset condition, the error correction method is to determine the target candidate character based on the stroke similarity between the target character and each candidate character in the candidate character set, as well as the probability of each candidate character, and then replacing the target character with the target candidate character; the weighted sum of the stroke similarity and probability of the target candidate character is maximized; the preset condition is that the character to be confirmed is the same as the target character, and the highest probability is greater than a preset probability threshold.

2. The text correction method according to claim 1, characterized in that, Obtaining the text sequence corresponding to the text to be corrected, the confidence score of each character in the text sequence, and the image sequence composed of the images of each character, including: The text image to be recognized is identified based on an optical character recognition model, and the text sequence, the confidence score of each character in the text sequence, and the coordinate value of each character are output. Based on the coordinate values ​​of each character, the image of the text to be identified is segmented to obtain the image sequence.

3. The text correction method according to claim 1, characterized in that, Based on the text sequence, the semantic information features corresponding to the text to be corrected are obtained, including: The text sequence is pre-encoded; The pre-encoded text sequence is input into the semantic information extractor of the pre-trained error correction model to obtain the semantic information features.

4. The text correction method according to claim 1, characterized in that, Based on the image sequence, the image information features corresponding to the text to be corrected are obtained, including: Each image in the image sequence is converted into a matrix of a preset size; The matrices are concatenated according to the order of the images in the image sequence, and the concatenated matrix is ​​input into the image feature extractor of the pre-trained error correction model to obtain the image information features.

5. The text correction method according to claim 1, characterized in that, The semantic information features and the image information features are fused, and based on the fused features, a set of candidate characters at each text position in the text to be corrected is predicted, including: Based on semantic information weights and image information weights, the semantic information features and the image information features are weighted and summed to obtain the fused features. The fused features are input into a normalized exponential function to predict the probability that each candidate character in the pre-built dictionary belongs to each of the character positions. For each of the text positions, multiple candidate characters are determined from the pre-established dictionary in descending order of probability value, forming the candidate character set.

6. The text correction method according to claim 5, characterized in that, Before performing a weighted summation of the semantic information features and the image information features based on semantic information weights and image information weights to obtain the fused features, the method further includes: The semantic information features and the image information features are concatenated, and a weight matrix is ​​constructed based on the concatenated features and preset weight parameters; The weight matrix is ​​solved using an activation function to output the semantic information weights and the image information weights.

7. The text correction method according to claim 1, characterized in that, For target texts in the text sequence with a confidence level less than a preset confidence threshold, error correction is performed based on a set of candidate characters at the text position of the target text, and further includes: If the character to be confirmed meets the preset conditions, then the target character is determined to be the correct character.

8. A text correction device, characterized in that, include: The acquisition module is used to: acquire the text sequence corresponding to the text to be corrected, the confidence level of each character in the text sequence, and the image sequence composed of the images of each character; The determining module is used to: obtain the semantic information features corresponding to the text to be corrected based on the text sequence, and obtain the image information features corresponding to the text to be corrected based on the image sequence; The semantic information features and the image information features are obtained through the semantic information extractor and image feature extractor of the pre-trained error correction model, respectively. The error correction model further includes a first fully connected layer, a transformer layer, and a second fully connected layer connected in sequence. The first fully connected layer is used to perform a linear transformation on the semantic information features and the image information features to obtain the weight information of the semantic information features and the image information features, respectively. The prediction module is used to fuse the semantic information features and the image information features, and based on the fused features, predict the candidate character set at each text position in the text to be corrected; the transformer layer is used to perform sentence-level understanding and fusion of the semantic information features and the image information features to obtain the fused features; the fused features are processed by the second fully connected layer and used to obtain the probability distribution of each text position belonging to a certain character; The error correction module is used to correct target characters in the text sequence whose confidence level is less than a preset confidence threshold, based on a set of candidate characters at the position of the target character. This includes: identifying the candidate character with the highest probability in the candidate character set as the character to be confirmed; if the character to be confirmed does not meet a preset condition, the error correction method is to determine the target candidate character based on the stroke similarity between the target character and each candidate character in the candidate character set, as well as the probability of each candidate character, and then replacing the target character with the target candidate character; the weighted sum of the stroke similarity and probability of the target candidate character is maximized; the preset condition is that the character to be confirmed is the same as the target character, and the highest probability is greater than a preset probability threshold.

9. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the method of any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images