Misspelling detection method, apparatus, device, and storage medium

Abstract

The application discloses a method and device for detecting misspelt words, an equipment and a storage medium. For a picture to be detected, a text block picture in which Chinese text is located is determined, recognized text contained in the text block picture is determined, image visual features of the text block picture are extracted, and character modal features of the recognized text are extracted. The image visual features and the character modal features are fused, real text contained in the text block picture is determined based on the fused features, and the real text is compared with the recognized text to obtain a misspelt word detection result. When detecting misspelt words in a picture, the application further fuses image visual features of a text block picture on the basis of considering character modal features of recognized text, so that the prediction result is more accurate. On this basis, a misspelt word detection result is determined by comparing the real text with the recognized text, and the accuracy of misspelt word detection is greatly improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a method, apparatus, device, and storage medium for detecting typos. Background Technology

[0002] With the development of information technology, an era characterized by diversified information transmission methods has arrived. In daily life and work, people receive more and more textual information from various sources, such as street advertisements and social media posts. At the same time, due to the accelerated pace of society, most widely disseminated textual content is becoming fragmented and simplified, such as public service announcements, posters, and product descriptions in advertising materials. Because this textual content is often incomplete, numerous, and disorganized, proofreading and correcting it requires a significant amount of manpower and time.

[0003] In today's era of booming artificial intelligence, especially thanks to advancements in natural language processing technology, various text error detection and correction systems have emerged to help people efficiently check and correct textual errors. The basic process of existing error correction systems is to receive a text segment as input, which may contain various errors such as grammatical and vocabulary errors; process it; locate and correct any existing errors; and return the location and correction results to the user. When correcting images, the process typically begins by recognizing the text in the image using methods such as OCR, and then simply correcting typos in the recognized text.

[0004] Existing error correction methods only utilize plain text information for error correction, resulting in low accuracy in typo detection. Summary of the Invention

[0005] In view of the above problems, this application is made to provide a method, apparatus, device, and storage medium for detecting typos, so as to improve the accuracy of typo detection. The specific solution is as follows:

[0006] Firstly, a method for detecting typos is provided, including:

[0007] Obtain the image to be detected that contains text;

[0008] Determine the text block image where the text in the image to be detected is located, and the recognized text contained in the text block image;

[0009] Extract the image visual features of the text block image, and extract the text modal features of the recognized text;

[0010] The image visual features and the text modal features are fused to obtain fused features;

[0011] Determine the real text contained in the text block image based on fusion features;

[0012] By comparing the real text and the recognized text, the typo detection results in the image to be detected are obtained.

[0013] Secondly, a typo detection device is provided, comprising:

[0014] The image acquisition unit is used to acquire images containing text to be detected;

[0015] The image processing unit is used to determine the text block image where the text in the image to be detected is located, and the recognized text contained in the text block image;

[0016] The feature extraction unit is used to extract the image visual features of the text block image, and to extract the text modal features of the recognized text;

[0017] The feature fusion unit is used to fuse the image visual features and the text modal features to obtain fused features;

[0018] The real text determination unit is used to determine the real text contained in a text block image based on fused features.

[0019] The misspelling detection unit is used to compare the real text and the recognized text to obtain the misspelling detection result in the image to be detected.

[0020] Thirdly, a typo detection device is provided, including: a memory and a processor;

[0021] The memory is used to store programs;

[0022] The processor is used to execute the program to implement the various steps of the above-described typo detection method.

[0023] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the above-described typo detection method.

[0024] By employing the aforementioned technical solution, this application identifies the text block image containing the text in the image to be detected, determines the recognized text contained within the text block image, further extracts the image visual features of the text block image, and extracts the text modal features of the recognized text. The image visual features and text modal features are then fused, and the real text contained within the text block image is determined based on the fused features. The real text and the recognized text are then compared to obtain the misspelling detection result. Therefore, this application, when detecting misspellings in images, considers the text modal features of the recognized text and further fuses the image visual features of the text block image. This fully utilizes the image visual features of the text block image to assist in predicting the real text, making the prediction result more accurate. Based on this, by comparing the real text and the recognized text, the misspelling detection result is determined, greatly improving the accuracy of misspelling detection. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0026] Figure 1 A flowchart illustrating the misspelling detection method provided in this application embodiment;

[0027] Figure 2 This example illustrates the process of marking typos in an image to be detected.

[0028] Figure 3 This example illustrates the structure of an image text recognition model.

[0029] Figure 4 An example is a schematic diagram of the structure of an image processing module;

[0030] Figure 5 This example illustrates a text block image annotation process.

[0031] Figure 6 This example illustrates the structure of a text processing module;

[0032] Figure 7 An example is a schematic diagram of the structure of a multimodal fusion module;

[0033] Figure 8 A schematic diagram illustrating the processing flow of a multimodal fusion module is provided.

[0034] Figure 9 This is a schematic diagram of a typo detection device provided in an embodiment of this application;

[0035] Figure 10 This is a schematic structural diagram of the typo detection device provided by the embodiments of the present application. Specific embodiments

[0036] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0037] The present application provides a typo detection method, which can be applicable to the typo detection tasks of text contained in various types of images, such as tasks of detecting typos in posters, advertisements, video screenshots, display screen images, etc.

[0038] The solution of the present application can be implemented based on a terminal with data processing capabilities, and the terminal can be a mobile phone, a computer, a server, the cloud, etc.

[0039] Next, in combination with Figure 1 As described above, the typo detection method of the present application may include the following steps:

[0040] Step S100: Obtain a to-be-detected picture containing text.

[0041] Specifically, the to-be-detected picture is a picture containing text, and the present application can detect typos in the text in the picture.

[0042] For the to-be-detected picture, it can be taken by the user, or can be a video screenshot, a screenshot of an electronic device, etc. The position of the text contained in the picture can be unrestricted. For example, it can be arranged in rows or columns.

[0043] The text contained in the to-be-detected picture can include Chinese and non-Chinese characters, such as English letters, special symbols, numbers, etc.

[0044] For example Figure 2 , which is a screenshot of an electronic device and contains subtitle text. It can be known that the character "根" in the subtitle "Art collection should not be blindly followed the trend" is a typo, and the correct one should be "跟".

[0045] Step S110: Determine the text block picture where the text in the to-be-detected picture is located, and the recognized text contained in the text block picture.

[0046] Specifically, in the embodiments of this application, an image text recognition algorithm can be used to identify the text block image in the image to be detected, such as using an OCR algorithm to identify the text block image and the identified text contained in the text block image.

[0047] It should be noted that the text block here can be a text line or a text column. For ease of description, the following explanation will use text lines as an example.

[0048] This step can involve either extracting a text block image from the image to be detected, or determining the position of the text block image within the image to be detected.

[0049] Depending on the distribution of text in the image to be detected, the number of text block images obtained in this step is unlimited, and can be one or more.

[0050] Step S120: Extract the image visual features of the text block image, and extract the text modal features of the recognized text.

[0051] Specifically, in this step, a pre-defined image vision algorithm can be used to extract the visual features of the text block image, or a pre-trained neural network model can be used to extract the visual features of the text block image. Image visual features describe the glyph information of the text from a visual perspective.

[0052] For the text block image containing the identified text, this step extracts the text modal features, also known as text features. Text modal feature extraction can be performed using a pre-defined text feature extraction algorithm or a pre-trained natural language processing model.

[0053] Step S130: Fuse the image visual features and the text modal features to obtain fused features.

[0054] Specifically, image visual features and text modal features describe relevant information from the perspectives of images and text, respectively. In order to more accurately predict the real text contained in the text block image, this step fuses the image visual features and text modal features, resulting in richer information and stronger expressive power in the fused features.

[0055] Step S140: Determine the real text contained in the text block image based on the fusion features.

[0056] Specifically, after obtaining the fused features in the above steps, the real text contained in the text block image can be predicted based on the fused features. This step can use a pre-trained neural network model to predict the real text.

[0057] The actual text predicted by this step is the correct text contained in the text block image identified in this application.

[0058] Step S150: Compare the real text and the recognized text to obtain the typos detection result in the to-be-detected picture.

[0059] Specifically, in this step, the real text can be used as a benchmark to compare the recognized text with the real text, determine whether the recognized text contains typos, and specifically determine the content of the typos contained, so as to obtain the typos detection result in the to-be-detected picture.

[0060] Exemplarily, in this step, it can be matched whether there are characters in the recognized text that are inconsistent with the real text. If so, the inconsistent characters in the recognized text are used as the typos in the to-be-detected picture.

[0061] The typos detection method provided by the embodiment of the present application determines the text block picture where the text is located in the to-be-detected picture, determines the recognized text contained in the text block picture, further extracts the image visual features of the text block picture, and extracts the text modality features of the recognized text, fuses the image visual features and the text modality features, and determines the real text contained in the text block picture based on the fused features, compares the real text with the recognized text to obtain the typos detection result. It can be seen that when detecting typos in a picture, the present application further fuses the image visual features of the text block picture on the basis of considering the text modality features of the recognized text, that is, fully utilizes the image visual features of the text block picture to assist in predicting the real text, making the prediction result more accurate. On this basis, by comparing the real text with the recognized text, the typos detection result is determined, greatly improving the accuracy of typos detection.

[0062] Optionally, after obtaining the typos detection result in the above step S150, if it is confirmed that the to-be-detected picture contains typos, the position of the typos in the to-be-detected picture can be further determined, and then the typos are marked in the to-be-detected picture according to the position, so as to visually display the typos in the to-be-detected picture.

[0063] Refer to Figure 2 , for the typo "root" recognized in the to-be-detected picture, it is marked in the form of a rectangular box.

[0064] Of course, the marking form of the typos is not limited to the rectangular box marking, and other various marking methods can also be adopted, such as highlighting, underlining, etc.

[0065] In this embodiment, the process of determining the position of the typos in the to-be-detected picture may specifically include:

[0066] First, determine the first position information of the text block image containing the misspelled word. This first position information is the position information of the text block image in the image to be detected.

[0067] Further determine the sorting order of misspelled words in the recognized text contained in the text block image.

[0068] Based on the first position information, the position of the first character in the text block image is determined in the image to be detected. According to the estimated width of each character, a sliding offset is used to shift the position of the first character backward by the width of the sorted characters in order, so as to locate the position of the misspelled word in the image to be detected.

[0069] In some embodiments of this application, the process of fusing the image visual features and the text modal features in step S130 to obtain fused features is described.

[0070] Optionally, the image visual features and text modal features extracted in step S120 can be in vector form. The vector dimensions of the image visual features and text modal features can be the same or different. Based on this, when performing feature fusion in this step, the two vector features can be fused to obtain fused features.

[0071] When performing vector fusion, various fusion methods can be used. In this embodiment, a gated fusion method is provided to fuse image visual features and text modal features in vector form to obtain fused features.

[0072] By employing a gated fusion method, using image visual features as the gate, partial features are extracted from the text modality features to obtain fused features. In other words, from the perspective of image visual features, the most important part of the text modality features is extracted as the feature representation of the fusion of image visual modality and character modality.

[0073] Optionally, this application provides several different gating fusion methods, such as: bitwise multiplication gating fusion, bitwise addition or division gating fusion, etc. For ease of description, the following embodiments use bitwise multiplication gating fusion as an example.

[0074] Furthermore, to avoid the loss of global features at the text language level, in this embodiment, the above-mentioned fusion features can be added to the text modal features to obtain residual fusion features, which are used as the final fusion features.

[0075] To enhance the richness of the image visual feature representation, before feature fusion in step S130, the image visual features can be subjected to representation shift and nonlinear transformation to obtain processed image visual features, which can then be fused with text modal features in step S130.

[0076] In some embodiments of this application, steps S120-S140 described in the foregoing embodiments can be obtained by processing a pre-trained image text recognition model.

[0077] For image-to-text recognition models, the configuration can be as follows: extract the visual features of the text block image, extract the text modal features of the text to be recognized in the text block image, fuse the visual features and text modal features, and predict the internal state representation of the real text contained in the text block image based on the fused features.

[0078] The input to the image text recognition model can include a text block image extracted from the image to be detected, and the recognition text contained in the text block image.

[0079] Alternatively, the input to the image text recognition model may include the image to be detected, the location information of the text block image within the image to be detected, and the recognized text contained in the text block image.

[0080] In this embodiment, by pre-training the image text recognition model, the powerful learning ability of the neural network model can be utilized to extract the image visual features of the text block image and the text modal features of the recognized text. Based on this, the real text is predicted after fusion.

[0081] Next, combined Figure 3 As shown, this embodiment provides an optional component structure for an image text recognition model.

[0082] An image text recognition model may include an image processing module, a text processing module, a multimodal fusion module, and an output module. Among them:

[0083] The image processing module is used to extract the visual features of text block images.

[0084] Specifically, if the input to the image text recognition model is a text block image, the image processing module can extract the image visual features from the text block image. If the input to the image text recognition model is the image to be detected and the position information of the text block images within it, the image processing module can first locate the text block images in the image to be detected based on the position information of the text block images, and then extract the image visual features from the text block images.

[0085] The text processing module is used to extract the text modal features of the recognized text contained in the text block image.

[0086] The multimodal fusion module is used to fuse the image visual features and the text modal features to obtain fused features.

[0087] The output module is used to determine the real text contained in the text block image based on the fusion features.

[0088] The output module can be trained using the MLM (Masked Language Model) method. Based on the fusion features output by the multimodal fusion module, it predicts the real text contained in the text block image.

[0089] Next, each of the above modules will be explained in detail.

[0090] 1. Image processing module

[0091] This embodiment describes an optional component structure of the image processing module, such as... Figure 4 As shown, it may include:

[0092] The image normalization module is used to normalize text block images to obtain processed text block images.

[0093] The input to the image normalization module can be the location information of the image to be detected and the text block image, or the text block image extracted from the image to be detected. Figure 4 In this example, we will use the first type of input as an example.

[0094] Because text block images in the real world vary in shape, such as different angles to the horizontal line and the property of objects appearing larger when closer and smaller when farther away, in order to better adapt to subsequent modules for feature extraction and ensure the quality of image visual feature extraction, this step can use an image standardization module to standardize the text block image. This can be done through algorithms such as image rotation, stretching, and scaling to process distorted text block images. The processed text block image has a set size, such as a matrix of size [96, 384].

[0095] For example Figure 5 The image above shows the original tilted and distorted text block image. After normalization, it becomes the text block image below.

[0096] The image feature extraction module is used to extract visual features from the processed text block image.

[0097] like Figure 4For example, an image feature extraction module can be composed of several visual feature recognition blocks connected in series. Each visual feature recognition block can include several convolutional layers, batch normalization layers, and nonlinear layers. The size and number of convolutional kernels in the convolutional layers within different visual feature recognition blocks can vary to enrich the perspectives of image visual feature extraction, thereby resulting in a richer and more accurate representation of the final image visual features.

[0098] The linear transformation module is used to perform a linear transformation on the dimensions of the image visual features to output image visual features with the same dimensions as the text modal features.

[0099] Specifically, the number of channels of the image visual features extracted by the image feature extraction module may not be directly matched with the dimension of the text modal features extracted by the text processing module. Therefore, it is necessary to perform a linear transformation on the dimension of the image visual features through the linear transformation module to output image visual features with the same dimension as the text modal features.

[0100] 2. Text Processing Module

[0101] This embodiment describes an optional structural configuration for the text processing module, such as... Figure 6 As shown, it may include:

[0102] The text preprocessing module is used to edit the recognized text in the text block image to a set length by filling it with a set character padding, and to determine the feature representation of the edited recognized text.

[0103] Specifically, to standardize the length of the recognized text in different text block images, this embodiment uses a text preprocessing module to edit the recognized text to a set length using padding. For recognized text shorter than the set length, a set padding character, such as [PAD], can be added to the end of the recognized text to supplement it to the set length. For recognized text longer than the set length, the set length can be truncated from the first character as a single edited recognized text. If the remaining length still exceeds the set length, the truncating operation is repeated. If the remaining length does not exceed the set length, the remaining portion is used as another edited recognized text.

[0104] For each edited piece of recognized text, a pre-trained tokenizer can be used to encode the text into a feature representation that the model can recognize. Specifically, the edited recognized text is segmented into words, and each word is encoded to obtain the token feature representation corresponding to the word.

[0105] Among them, the pre-trained tokenizer can adopt pre-trained model structures such as BERT tokenizer.

[0106] The text modality feature extraction module is used to encode the feature representation of the identified text to obtain the text modality features of the identified text.

[0107] Specifically, the text modality feature extraction module can use a pre-trained model (such as BERT, Transformer, etc.) to encode the feature representation of the recognized text after it has been processed by the text preprocessing module, so as to obtain the text modality features of the text block image.

[0108] 3. Multimodal fusion module

[0109] This embodiment describes an optional structural composition of the multimodal fusion module, such as... Figure 7 As shown, it may include: a feature editing module, a gating fusion module, and a residual connection module.

[0110] The processing flow of each module is combined Figure 8 Explanation:

[0111] The feature editing module is used to perform representation shift and nonlinear transformation on the visual features of the image to obtain the processed visual features of the image.

[0112] To enhance the representation of visual features in images, representation shifting and nonlinear transformations can be applied. Representation shifting involves adding a learnable bias parameter to each location of the visual feature. Nonlinear transformation uses nonlinear function layers, such as ReLU, sigmoid, or tanh layers, to nonlinearly transform the shifted visual features to a relatively small range near 0. For example, the range of a sigmoid transformation is (0, 1), and the range of a tanh transformation is (-1, 1).

[0113] The gated fusion module is used to fuse the visual features of the processed image and the text modal features using a gated fusion method to obtain fused features.

[0114] Specifically, in this embodiment, a gating fusion module is designed to perform bitwise multiplication, bitwise addition, or bitwise division to fuse the visual features and text modal features of the processed image to obtain fused features.

[0115] Figure 8 Taking the bitwise multiplication gated fusion method as an example, by using the bitwise multiplication gated fusion method, the visual features of the image are used as the gate to extract some features from the text modality features to obtain the fused features. That is, from the perspective of the visual features of the image, the most important part of the text modality features is extracted as the feature representation of the fusion of the visual modality and the character modality.

[0116] After the image visual features are processed by the feature editing module, they gain additional representational offsets and nonlinear transformations compared to text modal features. This maps the image visual features to a relatively small range near 0, such as the sigmoid function's range of (0,1). The range and distribution of text modal features remain unchanged. To put it figuratively, each position in the image visual features after editing is like a faucet (fully open corresponds to the upper bound of the nonlinear function's range, fully closed corresponds to the lower bound), controlling the information at the corresponding position in the text modal features. The more open the faucet is at a position in the image visual features, the more information is retained in the corresponding position in the text modal features; conversely, less is retained. Clearly, this positional multiplication yields the text modal feature portion, with the degree of retention controlled from a visual perspective. In other words, from the perspective of image visual features, the most important part of the text modal features is extracted as the feature representation for the fusion of image visual modality and character modality.

[0117] The residual connection module is used to add the fused feature to the text modal feature to obtain the residual fused feature, which is used as the final fused feature.

[0118] Furthermore, to avoid the loss of global features at the text language level, this embodiment can also add the above-mentioned fused features to the text modal features through the residual connection module to obtain residual fused features, which are used as the final fused features.

[0119] In some embodiments of this application, in order to further improve the accuracy of misspelling detection, after step S150, comparing the real text and the identified text to obtain the misspelling detection result in the image to be detected, a post-processing operation for misspelling verification can be further added.

[0120] In this embodiment, the post-processing for misspelling verification can be performed from the perspective of sentence semantic fluency, specifically including:

[0121] S1. Delete the typos identified in the identified text to obtain the edited text with the typos removed.

[0122] S2. Using a pre-trained language model, calculate the perplexity of the identified text and the edited text after deleting typos.

[0123] Specifically, perplexity is an indicator that measures the semantic fluency of a sentence; the more fluent a sentence is semantically, the lower its perplexity.

[0124] A language model is a probabilistic model used to calculate the probability that a sentence is a semantically correct sentence. Perplexity is a sentence-length-normalized metric related to the probability that a language model predicts a sentence. For a perfectly correct sentence, the lower the perplexity of the language model, the better the language model. Conversely, if a very good language model has been selected, then for a given sentence, if the perplexity of the language model is very low, it means that the sentence is highly likely to be correct.

[0125] In this step, to verify whether the previously identified typos are indeed typos, the perplexity of the identified text and the edited text after deleting the typos are calculated separately.

[0126] S3. If the perplexity of the edited text after deleting the typos is less than the perplexity of the recognized text, and the absolute value of the difference between the two is greater than a set threshold, then the typos are taken as the final typo detection result; otherwise, the typos are removed from the final typo detection result.

[0127] Understandably, if the perplexity of the edited text after deleting the typo is less than that of the recognized text, and the absolute value of the difference is greater than a set threshold, it indicates that the semantics of the edited text after deleting the typo is more coherent than that of the recognized text before deletion. In other words, the deleted typo was indeed a typo, and therefore it can be added to the final typo detection result. Conversely, if the perplexity is less than a set threshold, it indicates that the typo identified in the previous steps is a pseudo-typo, and it can be removed from the final typo detection result, meaning it will not be identified as a typo in the end.

[0128] In this embodiment, the accuracy of typo recognition in images is further improved by adding a post-processing operation that performs secondary verification of typos from the perspective of sentence semantic fluency.

[0129] The misspelling detection device provided in the embodiments of this application is described below. The misspelling detection device described below can be referred to in correspondence with the misspelling detection method described above.

[0130] See Figure 9 , Figure 9 This is a schematic diagram of the structure of a typo detection device disclosed in an embodiment of this application.

[0131] like Figure 9 As shown, the device may include:

[0132] Image acquisition unit 11 is used to acquire images to be detected that contain text;

[0133] Image processing unit 12 is used to determine the text block image where the text in the image to be detected is located, and the recognized text contained in the text block image;

[0134] The feature extraction unit 13 is used to extract the image visual features of the text block image and to extract the text modal features of the recognized text.

[0135] The feature fusion unit 14 is used to fuse the image visual features and the text modal features to obtain fused features;

[0136] The real text determination unit 15 is used to determine the real text contained in the text block image based on the fusion features;

[0137] The misspelling detection unit 16 is used to compare the real text and the recognized text to obtain the misspelling detection result in the image to be detected.

[0138] Optionally, if the image visual features and the text modal features are both in vector form, then the process by which the feature fusion unit fuses the image visual features and the text modal features to obtain fused features may include:

[0139] By employing a gated fusion method, the visual features of the image in vector form and the modal features of the text are fused to obtain the fused features.

[0140] Optionally, the above gating fusion methods may include gating fusion methods of bitwise multiplication, bitwise addition or division, etc.

[0141] Optionally, after fusing the image visual features and text modal features in vector form using a gated fusion method, the feature fusion unit may further include:

[0142] The fused features are added to the text modal features to obtain residual fused features, which are used as the final fused features.

[0143] Optionally, before fusing the vector-based image visual features and text modal features using a gated fusion method, the aforementioned feature fusion unit may further include:

[0144] The image visual features are subjected to representation shift and nonlinear transformation to obtain the processed image visual features.

[0145] Optionally, the processing of the above-mentioned feature extraction unit 13, feature fusion unit 14 and real text determination unit 15 can be implemented by a pre-trained image text recognition model. The image text recognition model is configured to extract the image visual features of the text block image, extract the text modal features of the recognized text in the text block image, fuse the image visual features and the text modal features, and predict the internal state representation of the real text contained in the text block image based on the fused features.

[0146] The image text recognition model may include: an image processing module, a text processing module, a multimodal fusion module, and an output module;

[0147] The image processing module is used to extract the visual features of the text block image;

[0148] The text processing module is used to extract the text modal features of the recognized text contained in the text block image;

[0149] A multimodal fusion module is used to fuse the image visual features and the text modal features to obtain fused features;

[0150] The output module is used to determine the real text contained in the text block image based on the fusion features.

[0151] Optionally, the above-mentioned multimodal fusion module may further include:

[0152] The feature editing module is used to perform representation shift and nonlinear transformation on the visual features of the image to obtain the processed visual features of the image.

[0153] The gated fusion module is used to fuse the visual features of the processed image and the text modal features using a gated fusion method to obtain fused features;

[0154] The residual connection module is used to add the fused feature to the text modal feature to obtain the residual fused feature, which is used as the final fused feature.

[0155] Optionally, the image processing module described above may further include:

[0156] The image standardization module is used to standardize text block images to obtain processed text block images.

[0157] The image feature extraction module is used to extract image visual features from the processed text block image;

[0158] The linear transformation module is used to perform a linear transformation on the dimensions of the image visual features to output image visual features with the same dimensions as the text modal features.

[0159] Optionally, the above text processing module may further include:

[0160] The text preprocessing module is used to edit the recognized text in the text block image to a set length by filling in set characters, and to determine the feature representation of the edited recognized text.

[0161] The text modality feature extraction module is used to encode the feature representation of the identified text to obtain the text modality features of the identified text.

[0162] Optionally, the process by which the above-mentioned misspelling determination unit compares the real text and the recognized text to obtain the misspelling detection result in the image to be detected may include:

[0163] The system checks whether there are any characters in the identified text that are inconsistent with the real text. If so, the inconsistent characters in the identified text are considered as typos in the image to be detected.

[0164] Optionally, the apparatus of this application may further include: a misspelling verification unit, configured to: after comparing the real text and the recognized text to obtain the misspelling detection result in the image to be detected, delete the misspellings identified in the recognized text to obtain the recognized text with the misspellings removed; use a pre-trained language model to calculate the perplexity of the recognized text and the recognized text with the misspellings removed respectively; if the perplexity of the recognized text with the misspellings removed is less than the perplexity of the recognized text, and the absolute value of the difference between the two is greater than a set threshold, then the misspelling is taken as the final misspelling detection result; otherwise, the misspelling is removed from the final misspelling detection result.

[0165] Optionally, the apparatus of this application may further include: a misspelling marking unit, configured to: after comparing the real text and the recognized text to obtain a misspelling detection result in the image to be detected, determine the position of the misspelling in the image to be detected; and mark the misspelling in the image to be detected according to the position.

[0166] The typo detection device provided in this application embodiment can be applied to typo detection equipment, such as terminals: mobile phones, computers, etc. Optionally, Figure 10 The hardware structure block diagram of the typo detection device is shown below. Figure 10 The hardware structure of a typo detection device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0167] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0168] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0169] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0170] The memory stores a program, which the processor can call. The program is used for:

[0171] Obtain the image to be detected that contains text;

[0172] Determine the text block image where the text in the image to be detected is located, and the recognized text contained in the text block image;

[0173] Extract the image visual features of the text block image, and extract the text modal features of the recognized text;

[0174] The image visual features and the text modal features are fused to obtain fused features;

[0175] Determine the real text contained in the text block image based on fusion features;

[0176] By comparing the real text and the recognized text, the typo detection results in the image to be detected are obtained.

[0177] Optionally, the refined and extended functions of the program can be found in the description above.

[0178] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0179] Obtain the image to be detected that contains text;

[0180] Determine the text block image where the text in the image to be detected is located, and the recognized text contained in the text block image;

[0181] Extract the image visual features of the text block image, and extract the text modal features of the recognized text;

[0182] The image visual features and the text modal features are fused to obtain fused features;

[0183] Determine the real text contained in the text block image based on fusion features;

[0184] By comparing the real text and the recognized text, the typo detection results in the image to be detected are obtained.

[0185] Optionally, the refined and extended functions of the program can be found in the description above.

[0186] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0188] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of detecting misspelt words, characterized by, include: Obtain the image to be detected that contains text; The text block image containing the text in the image to be detected is determined, as well as the recognized text contained in the text block image; the text block includes: text lines or text columns; Extract the image visual features of the text block image, and extract the text modal features of the recognized text; The image visual features and the text modal features are fused to obtain fused features, including: using a gated fusion method, the image visual features are used as a gate to extract a portion of the text modal features as a feature representation of the fusion of image visual modality and character modality, thereby obtaining fused features; Determine the real text contained in the text block image based on fusion features; By comparing the real text and the recognized text, the typo detection results in the image to be detected are obtained.

2. The method of claim 1, wherein, The image visual features and the text modal features are both in vector form; The process of using a gated fusion method, where the image visual features are used as a gate, to extract a portion of the text modality features as the feature representation for the fusion of the image visual modality and the character modality, and to obtain the fused features, includes: A gated fusion approach is adopted, using image visual features in vector form as the gate, and extracting some features from text modality features in vector form as the feature representation for the fusion of image visual modality and character modality, thus obtaining the fused features.

3. The method of claim 2, wherein, After employing a gated fusion method, using vector-based image visual features as the gate, and extracting a portion of the vector-based text modality features as the feature representation for the fusion of image visual modality and character modality, the resulting fused features also include: The fused features are added to the text modal features to obtain residual fused features, which are used as the final fused features.

4. The method of claim 2, wherein, Before obtaining the fused features, the following steps are included: Using a gated fusion approach, the image visual features in vector form are used as the gate, and a portion of the text modality features in vector form are extracted as the feature representation for the fusion of the image visual modality and the character modality. The image visual features are subjected to representation shift and nonlinear transformation to obtain the processed image visual features.

5. The method of claim 1, wherein, The process of extracting the visual features and text modal features of the image and fusing them, and determining the real text contained in the text block image based on the fused features, is obtained through a pre-trained image text recognition model. The image-text recognition model is configured to extract the visual features of the text block image, extract the text modal features of the recognized text in the text block image, fuse the visual features and text modal features, and predict the internal state representation of the real text contained in the text block image based on the fused features.

6. The method of claim 5, wherein, The image text recognition model includes: an image processing module, a text processing module, a multimodal fusion module, and an output module; The image processing module is used to extract the visual features of the text block image. The text processing module is used to extract the text modal features of the recognized text contained in the text block image; A multimodal fusion module is used to fuse the image visual features and the text modal features to obtain fused features; The output module is used to determine the real text contained in the text block image based on the fusion features.

7. The method of claim 6, wherein, The multimodal fusion module includes: The feature editing module is used to perform representation shift and nonlinear transformation on the visual features of the image to obtain the processed visual features of the image. The gated fusion module is used to fuse the visual features of the processed image and the text modal features using a gated fusion method to obtain fused features; The residual connection module is used to add the fused feature to the text modal feature to obtain the residual fused feature, which is used as the final fused feature.

8. The method of claim 6, wherein, The image processing module includes: The image standardization module is used to standardize text block images to obtain processed text block images. The image feature extraction module is used to extract image visual features from the processed text block image; The linear transformation module is used to perform a linear transformation on the dimensions of the image visual features to output image visual features with the same dimensions as the text modal features.

9. The method of claim 6, wherein, The text processing module includes: The text preprocessing module is used to edit the recognized text in the text block image to a set length by filling in set characters, and to determine the feature representation of the edited recognized text. The text modality feature extraction module is used to encode the feature representation of the identified text to obtain the text modality features of the identified text.

10. The method according to any one of claims 1 to 9, characterized in that, The process of comparing the real text and the recognized text to obtain the typo detection result in the image to be detected includes: The system checks whether there are any characters in the identified text that are inconsistent with the real text. If so, the inconsistent characters in the identified text are considered as typos in the image to be detected.

11. The method according to any one of claims 1 to 9, characterized in that, After comparing the real text and the recognized text to obtain the typo detection results in the image to be detected, the method further includes: The identified typos in the text are deleted to obtain the edited text after removing the typos; Using a pre-trained language model, the perplexity of the identified text and the edited text after deleting typos is calculated respectively; If the perplexity of the edited text after deleting the typos is less than the perplexity of the recognized text, and the absolute value of the difference between the two is greater than a set threshold, then the typo is taken as the final typo detection result; otherwise, the typo is removed from the final typo detection result.

12. The method according to any one of claims 1 to 9, characterized in that, After comparing the real text and the recognized text to obtain the typo detection results in the image to be detected, the method further includes: Determine the location of the misspelled word in the image to be detected; The misspelled words are marked in the image to be detected according to their positions.

13. A misspelling detection apparatus characterized by comprising: include: The image acquisition unit is used to acquire images containing text to be detected; The image processing unit is used to determine the text block image in the image to be detected, and the recognized text contained in the text block image; the text block includes: text lines or text columns; The feature extraction unit is used to extract the image visual features of the text block image, and to extract the text modal features of the recognized text; The feature fusion unit is used to fuse the image visual features and the text modal features to obtain fused features, including: using a gated fusion method, using the image visual features as a gate, extracting a portion of the features from the text modal features as a feature representation of the fusion of image visual modality and character modality to obtain fused features; The real text determination unit is used to determine the real text contained in a text block image based on fused features. The misspelling detection unit is used to compare the real text and the recognized text to obtain the misspelling detection result in the image to be detected.

14. A misspelling detection device characterized by comprising: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the misspelling detection method as described in any one of claims 1 to 12.

15. A storage medium having stored thereon a computer program, characterized in that When the computer program is executed by the processor, it implements each step of the misspelling detection method as described in any one of claims 1 to 12.

Images