Semantic Analysis-Based Image Recognition and Restoration Methods, Systems, Equipment, and Media

Abstract

This application discloses a semantic analysis-based image-text recognition and repair method, system, device, and medium, comprising: acquiring a first text image to be recognized, processing the first text image to obtain a second text image; passing the second text image through a pre-established text box detection model to obtain text boxes of the second text image; passing the text boxes of the second text image through a pre-constructed character recognition model to obtain a first character recognition result; adding a language model based on the first character recognition result to analyze whether the text boxes can be connected, and splitting and recombining the text between different text boxes to ensure the integrity of the text data within the same unit, finally obtaining a second character recognition result. This application incorporates an NLP text correction task into a traditional character recognition model and adds a language model to the character recognition model to improve the accuracy of text recognition and obtain complete text data.

Description

Technical Field

[0001] This application relates to the field of text and image recognition technology, specifically to a text and image recognition and restoration method, system, device, and medium based on semantic analysis. Background Technology

[0002] In production and daily life, we often need to process large amounts of text, reports, or documents. Text recognition technology can reduce people's workload and improve work efficiency. Current methods for processing text images are as follows: Method 1: For the input text image, the DB algorithm is used to detect and locate the text within the image, and then the CRNN algorithm is used to perform character recognition on the detected text boxes, outputting the text content contained in the image; Method 2: The DB algorithm is used to detect and locate text boxes in the text image, and the SVTR algorithm is used to train a character recognition model for character recognition on the detected text boxes, finally outputting the text content contained in the image.

[0003] The method described in Method 1 has a simple model structure, is easy to train, and converges easily, but its character recognition accuracy is quite low. When dealing with a large number of Chinese characters, it performs significantly worse than RARE-type attention models, and even worse with English characters. For short texts with significant distortions, such as artistic fonts, or texts with large variations in natural scenes, this method's accuracy is even lower. Method 2 addresses some of the problems of Method 1, but its accuracy decreases in complex scenes, such as uneven lighting, high background noise, and blurred text. Furthermore, it performs poorly with long texts, multilingual texts, and multi-font texts. Due to the significant differences in character shapes and structures across different languages ​​and fonts, SVTR requires more training data and algorithm optimization to achieve better recognition results. Therefore, it is necessary to improve the accuracy of long texts and multi-font text image recognition in complex situations. Summary of the Invention

[0004] The main purpose of this application is to overcome the shortcomings and deficiencies of the existing technology and provide a semantic analysis-based image and text recognition and repair method, system, device and medium. By adding NLP text correction tasks to the traditional character recognition model and adding a language model to the character recognition model, the accuracy of text recognition can be improved and complete text data can be obtained.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, this application provides a text-image recognition and restoration method based on semantic analysis, comprising the following steps:

[0007] Obtain the first text image to be recognized, and process the first text image to obtain the second text image;

[0008] The second text image is processed through a pre-defined text box detection model to obtain the text box of the second text image;

[0009] The text box of the second text image is used to obtain the first character recognition result through a pre-built character recognition model;

[0010] Based on the first character recognition result, a language model is added to analyze whether the text boxes can be connected, and the text between different text boxes is split and recombined to make the text data in the same cell complete, and finally the second character recognition result is obtained.

[0011] As a preferred technical solution, the character recognition model includes a patch embedding module, a feature mixing module, a merging module, an error character localization module, and a character correction network module. The patch embedding module is used to convert a two-dimensional image into a series of one-dimensional patch blocks. The feature mixing module is used to extract character feature sequences. The merging module is used to generate a transcription sequence from the character feature sequences. The error character localization module is used to determine the location information of error characters. The character correction network module is used to correct the transcribed character sequences. The feature mixing module further includes a global mixing module and a local mixing module. The global mixing module is used to evaluate the dependencies between all character components and establish dependencies between different character components. The local mixing module is used to simulate stroke example features.

[0012] As a preferred technical solution, the step of obtaining the first character recognition result by passing the text box of the second text image through a pre-built character recognition model includes the following steps:

[0013] The patch embedding module performs a 1 / 4 downsampling on the second text image through convolution to obtain a feature map; the convolution uses the BN paradigm to standardize the data.

[0014] The feature fusion module performs height downsampling on the feature map to obtain a low-dimensional feature map;

[0015] The merging module first performs high-dimensional global pooling and then compresses the characters into a feature sequence through a fully connected layer, thereby generating a transcription sequence; identical characters are transcribed into repeated characters, and non-text components are transcribed into whitespace characters; at the same time, the first feature value of the character is calculated.

[0016] The error character module integrates the transcribed characters and compares them with the annotation results of the text box of the second text image to be recognized. It determines the characters that differ from the annotated characters and their position information, and calculates the second feature value of the error character.

[0017] The character correction network module first replaces the second feature value in the first feature value of the output with a mask to obtain the third feature value. The third feature value is then used to balance the first feature value of the string by multiplying the probability to obtain the fourth feature value. Next, the fourth feature value is used as the input to the Transformer module in the character correction network module. The output of the last layer of the Transformer module is residually connected with the first feature value to obtain the final feature of each character. Finally, the final feature of each character is input into the softmax classifier, which outputs the character with the highest probability from the candidate word list as the correct character.

[0018] As a preferred technical solution, the step of adding a language model based on the first character recognition result, analyzing whether the text boxes can be connected, and splitting and recombining the text between different text boxes to ensure the integrity of the text data within the same unit, ultimately obtaining the second character recognition result; the specific steps include:

[0019] Sort the coordinates of the text boxes in the second text image in ascending order, and check if there is a table in the second text image. If there is a table, mark the position of the table.

[0020] Merge text at the same height outside the table, as well as text at different heights;

[0021] Within the defined location range of the table, the table is divided into cells, and the text in the cells is reorganized to obtain the second character recognition result.

[0022] As a preferred technical solution, the steps for merging text at the same height outside the table and text at different heights are as follows:

[0023] Merging text at the same height outside the table:

[0024] Calculate the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference is within the threshold range, the two text boxes can be connected.

[0025] Merging text of different heights outside the table:

[0026] Calculate the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference exceeds a threshold, calculate the difference between the right coordinate of the text box and the global right coordinate of the second text image, and the difference between the left coordinate of the next text box and the global left coordinate of the second text image. If both differences are less than the threshold, merge the characters of the two text boxes and add them to the dictionary, part-of-speech logic judgment, and N-gram language model. Finally, calculate the probability that the text combined from the two text boxes becomes a sentence. If the probability is higher than the threshold, directly merge the text from the two text boxes.

[0027] As a preferred technical solution, the step of dividing the table into cells is as follows:

[0028] Within the defined table area, the super-resolution algorithm TSR is used to detect the horizontal and vertical lines in the table and calculate the intersection points of the lines to form individual cells, thus determining the structure of the table and the coordinates of each cell.

[0029] Based on a tree model, continuous learning is performed using CNN as a foundation. At each decision step, a SequentialAttention model is used to select useful features for inference and a TableNet model is constructed. The TableNet model is then used to analyze the cell coordinates in the table.

[0030] The final table structure is obtained by combining the coordinates of each cell.

[0031] As a preferred technical solution, the step of reorganizing the text in the cell to obtain the second character recognition result includes:

[0032] For the same text box, calculate the coordinates of the four sides of the text box and the difference between them and the cell boundary. If the difference exceeds the threshold, the text box is considered to span cells. Then, determine whether the text spanning cells is split in the direction of top, bottom, left, or right. Find the cells in the corresponding split direction, divide the text box into multiple parts using the cell boundary coordinates, and reconstruct the coordinates of the text box.

[0033] For each text box in the same cell, a language model is used to calculate the combination that scores the highest for forming a sentence from the text in each text box, and finally the text in the cell is output.

[0034] Secondly, this application provides a semantic analysis-based image and text recognition and repair system, which is applied to the semantic analysis-based image and text recognition and repair method, including a module for obtaining the image of the text to be recognized, a text box detection model module, a first character recognition result module, and a second character recognition result module;

[0035] The module for obtaining the text image to be recognized is used to obtain a first text image to be recognized and process the first text image to obtain a second text image.

[0036] The text box detection model module is used to obtain the text box of the second text image by passing it through a pre-established text box detection model;

[0037] The first character recognition result module is used to obtain the first character recognition result by passing the text box of the second text image through a pre-built character recognition model;

[0038] The second character recognition result module is used to add a language model based on the first character recognition result, analyze whether the text boxes can be connected, and split and reorganize the text between different text boxes so that the text data in the same cell is complete, and finally obtain the second character recognition result.

[0039] Thirdly, this application provides an electronic device, the electronic device comprising:

[0040] At least one processor; and,

[0041] A memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores computer program instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the semantic analysis-based image recognition and restoration method.

[0043] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, implements the semantic analysis-based image recognition and repair method.

[0044] In summary, compared with the prior art, the effective effects of the technical solution provided in this application include at least the following:

[0045] 1. This application incorporates NLP text correction elements into the traditional character recognition model. Since character recognition only considers the character features based on their shape and strokes and does not incorporate the semantic relationship between the preceding and following text, the addition of a correction network can take into account the preceding and following context and correct the incorrectly recognized characters.

[0046] 2. This application incorporates table detection and cell reconstruction algorithms to distinguish text into two types: text inside and outside the table, thereby reducing recognition noise and interference. Semantic analysis is added to the text box connection outside the table, which can greatly improve the problem of a sentence being divided into multiple text boxes and becoming incoherent, and reconstruct a complete sentence or a complete paragraph.

[0047] 3. For text data in the table, the text boxes in the table are split into specific cells, and the text in the cells is recombined using a language model to improve the recognition accuracy of text in the table. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a semantic analysis-based image recognition and restoration method according to an embodiment of this application.

[0050] Figure 2 This is a flowchart illustrating the operation of a semantic analysis-based image recognition and restoration method according to an embodiment of this application.

[0051] Figure 3 This is a block diagram of a semantic analysis-based image recognition and repair system according to an embodiment of this application. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0053] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0054] Please see Figure 1 , Figure 2 One embodiment of this application provides a text-image recognition and restoration method based on semantic analysis, comprising the following steps:

[0055] S1. Obtain the first text image to be recognized, and process the first text image to obtain the second text image;

[0056] Furthermore, after obtaining the first text image to be identified, it is determined whether the first text image to be identified has blurry, skewed, or shadowed parts. If so, the first text image is processed to improve its quality.

[0057] S2. The second text image is processed through a pre-established text box detection model to obtain the text box of the second text image;

[0058] Furthermore, the text box detection model is built using the DB algorithm. The model is trained to generate an inference model, which detects the text boxes in the second text image.

[0059] S3. The text box of the second text image is processed by a pre-built character recognition model to obtain the first character recognition result;

[0060] S31. Further, the character recognition model includes a patch embedding module, a feature mixing module, a merging module, an error character localization module, and a character correction network module; the patch embedding module is used to convert a two-dimensional image into a series of one-dimensional patch blocks; the feature mixing module is used to extract character feature sequences; the merging module is used to generate transcription sequences from the character feature sequences; the error character localization module is used to determine the location information of error characters; the character correction network module is used to correct the transcribed character sequences; wherein, the feature mixing module further includes a global mixing module and a local mixing module; the global mixing module is used to evaluate the dependencies between all character components and establish dependencies between different character components; the local mixing module is used to simulate stroke example features.

[0061] S32. The text box of the second text image is processed through a pre-built character recognition model to obtain the first character recognition result. The specific steps include:

[0062] S32-1, Patch Embedding Module: The patch embedding module downsamples the second text image by 1 / 4 through two convolutions to obtain the feature map. The convolution is a 3×3 convolution with a stride of 2, which is stacked with patch embedding. The convolution uses the BN paradigm to standardize the data.

[0063] S32-2, Feature Blending Module: The feature blending module performs height downsampling on the feature map to obtain a low-dimensional feature map;

[0064] Furthermore, the feature mixing module contains three downsampling modules, and all three downsampling modules only downsample the height of the feature map. This is because the wide dimension contains a lot of text information, and downsampling in this dimension would result in the loss of a lot of information.

[0065] Furthermore, the global fusion module in the feature fusion module is constructed by a self-attention mechanism, which can evaluate the dependencies between all character components. Since text and non-text are the two main elements in character recognition, this fusion method can establish dependencies between different character components, while also reducing the influence of non-text components and increasing the weight of text components.

[0066] The local blending block in the feature blending module is used to evaluate the correlation between components within a predefined window. Blending is performed in a sliding window manner. This blending block can simulate stroke sample features, which is very important for character recognition.

[0067] S32-3, Merging Module and Calculating First Feature Value: The merging module first performs high-dimensional global pooling, and then compresses the characters into a feature sequence through a fully connected layer, thereby generating a transcription sequence; the same characters are transcribed into repeated characters, and non-text components are transcribed into whitespace characters; at the same time, the first feature value of the character is calculated;

[0068] S32-4, Error Character Location Module: The error character module integrates the transcribed characters and compares them with the annotation results of the text box of the second text image to be identified, determines the characters that differ from the annotated characters and their position information, and calculates the second feature value of the error character;

[0069] S32-5, Character Correction Network Module:

[0070] First, the first feature value output in step S32-3 replaces the second feature value at the location of the erroneous character in step S32-4 with a mask, resulting in the third feature value. The third feature value is then used to balance the first feature value of the string through a probability product, resulting in the fourth feature value. The calculation formula is as follows:

[0071] e′ i =p i ·e mask +(1-p i )·e i

[0072] Among them, e′ i e is the fourth eigenvalue. mask e is the third eigenvalue. i p is the first eigenvalue. i This indicates the probability of an incorrect character at that position.

[0073] Finally, each character feature is passed through a softmax classifier, and the character with the highest probability from the candidate word list is considered the correct character for each position.

[0074] S4. Based on the first character recognition result, add a language model to analyze whether the text boxes can be connected, and split and recombine the text between different text boxes to ensure the integrity of the text data within the same unit, ultimately obtaining the second character recognition result; the specific steps include:

[0075] S41. Sort the coordinates of the text boxes of the second text image in ascending order, and check whether there is a table in the second text image. If there is a table, mark the position of the table.

[0076] S42. Merge text at the same height outside the table and text at different heights respectively;

[0077] S42-1. Merging text at the same height outside the table:

[0078] Calculate the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference is within the threshold range, the two text boxes can be connected.

[0079] S42-2, Merging text at different heights outside the table:

[0080] Calculate the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference exceeds a threshold, calculate the difference between the right coordinate of the text box and the global right coordinate of the second text image, and the difference between the left coordinate of the next text box and the global left coordinate of the second text image. If both differences are less than the threshold, merge the characters of the two text boxes and add them to the dictionary, part-of-speech logic judgment, and N-gram language model. Finally, calculate the probability that the text combined from the two text boxes becomes a sentence. If the probability is higher than the threshold, directly merge the text from the two text boxes.

[0081] S43. Within the defined position range of the table, the table is divided into cells, and the text of the cells is reorganized to obtain the second character recognition result.

[0082] S43-1. Further, the table is divided into cells, the steps are as follows:

[0083] (1) Within the defined table range, the super-resolution algorithm TSR is used to detect the horizontal and vertical lines in the table and calculate the intersection of the lines to form individual cells, and finally determine the structure of the table and the coordinates of each cell.

[0084] (2) Adopting the tree model idea, based on CNN for continuous learning, at each decision step, the SequentialAttention model is used to select some features for inference, because the sample also has a certain influence on the selection of features; and a TableNet model is constructed, through which the cell coordinates in the table can be analyzed;

[0085] Combining the results obtained from (1) and (2) above, the final table structure is obtained.

[0086] S43-2, The steps for reorganizing the text in the cell to obtain the second character recognition result include:

[0087] For the same text box, calculate the coordinates of the four sides of the text box and the difference between them and the cell boundary. If the difference exceeds the threshold, the text box is considered to span cells. Then, determine which directions (top, bottom, left, right) the text box is split in, find the cells in the corresponding split directions, divide the text box into multiple parts using the cell boundary coordinates, and reconstruct the coordinates of the text box.

[0088] For each text box within the same cell, a language model is used to calculate the combination that yields the highest score when the text in each text box forms a sentence. The final output is the text within that cell. The mathematical representation of the language model probability is as follows:

[0089]

[0090] Among them, P(w i |w1……w i-1 The probability of generating the i-th (1≤i≤m) word is determined by the probability of generating the i-1 words w1……w1. i-1 Determined; m represents the total number of words in the text box.

[0091] In summary, this application first incorporates NLP text correction elements into traditional character recognition models. Character recognition only considers the morphological strokes of a character to determine which character it should be transcribed into, without considering the semantic relationships between preceding and following text. Adding a correction network allows for consideration of the context and correction of incorrectly recognized characters. Second, directly connecting OCR recognition results by sorting text boxes results in text with significant noise, greatly complicating downstream tasks. This application incorporates table recognition to improve the accuracy of recognizing text both inside and outside tables. In project applications, OCR recognition is just the beginning; numerous tasks need to be completed, such as keyword extraction and key personnel factor extraction. The completeness of the text recognition results is a major obstacle to these tasks. Third, this application incorporates semantic analysis into the connection of text boxes outside tables, significantly improving the problem of disjointed text when a sentence is divided into multiple text boxes. Reconstructing a complete sentence or paragraph is crucial for text image recognition. This application further subdivides text boxes within tables into individual cells and utilizes NLP language models to reconstruct the text within cells, which is significant for achieving table reconstruction.

[0092] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously.

[0093] Based on the same idea as the semantic analysis-based image recognition and restoration method in the above embodiments, this application also provides a semantic analysis-based image recognition and restoration system, which can be used to execute the above-described semantic analysis-based image recognition and restoration method. For ease of explanation, the structural diagram of the semantic analysis-based image recognition and restoration system embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0094] Please see Figure 3 In another embodiment of this application, a semantic analysis-based image recognition and repair system 100 is provided. The system includes a module 101 for acquiring images of text to be recognized, a text box detection model module 102, a first character recognition result module 103, and a second character recognition result module 104.

[0095] The module 101 for obtaining the text image to be recognized is used to obtain a first text image to be recognized and process the first text image to obtain a second text image.

[0096] The text box detection model module 102 is used to obtain the text box of the second text image by passing the second text image through a pre-established text box detection model;

[0097] The first character recognition result module 103 is used to obtain the first character recognition result by passing the text box of the second text image through a pre-built character recognition model;

[0098] The second character recognition result module 104 is used to add a language model based on the first character recognition result, analyze whether the text boxes can be connected, and split and reorganize the text between different text boxes so that the text data in the same cell is complete, and finally obtain the second character recognition result.

[0099] It should be noted that the semantic analysis-based image and text recognition and restoration system of this application corresponds one-to-one with the semantic analysis-based image and text recognition and restoration method of this application. The technical features and beneficial effects described in the embodiments of the semantic analysis-based image and text recognition and restoration method described above are all applicable to the embodiments of the semantic analysis-based image and text recognition and restoration system. For details, please refer to the description in the embodiments of the method of this application, which will not be repeated here.

[0100] Furthermore, in the above embodiments of the semantic analysis-based image recognition and restoration system, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the semantic analysis-based image recognition and restoration system can be divided into different program modules to complete all or part of the functions described above.

[0101] In another embodiment, an electronic device is provided that implements a semantic analysis-based image and text recognition and restoration method, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, it implements the semantic analysis-based image and text recognition and restoration method of any embodiment of this application.

[0102] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the device.

[0103] The device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The device may include, but is not limited to, a processor and memory.

[0104] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the device, connecting various parts of the device via various interfaces and lines.

[0105] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory may include high-speed random access memory and non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0106] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the semantic analysis-based image recognition and repair method as described in any of the above embodiments.

[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above embodiments are preferred embodiments of this application, but the implementation of this application is not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of this application shall be considered equivalent substitutions and shall be included within the protection scope of this application.

Claims

1. A semantic analysis-based image-text recognition and restoration method, characterized in that, Includes the following steps: Obtain the first text image to be recognized, and process the first text image to obtain the second text image; The second text image is processed through a pre-defined text box detection model to obtain the text box of the second text image; The text box of the second text image is used to obtain the first character recognition result through a pre-built character recognition model; The character recognition model includes a patch embedding module, a feature mixing module, a merging module, an error character localization module, and a character correction network module; the patch embedding module is used to convert a two-dimensional image into a series of one-dimensional patch blocks; The feature mixing module is used to extract character feature sequences; the merging module is used to generate transcribed sequences from the character feature sequences; the error character localization module is used to determine the location information of error characters; the character correction network module is used to correct transcribed character sequences; wherein, the feature mixing module further includes a global mixing module and a local mixing module; the global mixing module is used to evaluate the dependencies between all character components and establish dependencies between different character components; the local mixing module is used to simulate stroke example features; Sort the coordinates of the text boxes in the second text image in ascending order, and check if there is a table in the second text image. If there is a table, mark the position of the table. Merging text at the same height outside the table involves calculating the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference is within a threshold range, the two text boxes can be connected. The text merging process outside the table at different heights involves calculating the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference exceeds a threshold, the process calculates the difference between the right coordinate of the text box and the global right coordinate of the second text image, and the difference between the left coordinate of the next text box and the global left coordinate of the second text image. If both differences are less than the threshold, the characters in the two text boxes are merged, and the dictionary, part-of-speech logic judgment, and N-gram language model are added. Finally, the probability that the combined text of the two text boxes forms a sentence is calculated. If the probability is higher than the threshold, the text in the two text boxes is directly merged. Within the defined location range of the table, the table is divided into cells, and the text in the cells is reorganized to obtain the second character recognition result.

2. The image-text recognition and restoration method based on semantic analysis according to claim 1, characterized in that, The specific steps for obtaining the first character recognition result by passing the text box of the second text image through a pre-built character recognition model include: The patch embedding module performs a 1 / 4 downsampling on the second text image through convolution to obtain a feature map; the convolution uses the BN paradigm to standardize the data. The feature fusion module performs height downsampling on the feature map to obtain a low-dimensional feature map; The merging module first performs high-dimensional global pooling and then compresses the characters into a feature sequence through a fully connected layer, thereby generating a transcription sequence; identical characters are transcribed into repeated characters, and non-text components are transcribed into whitespace characters; at the same time, the first feature value of the character is calculated. The error character module integrates the transcribed characters and compares them with the annotation results of the text box of the second text image to be recognized. It determines the characters that differ from the annotated characters and their position information, and calculates the second feature value of the error character. The character correction network module first replaces the second feature value in the first feature value of the output with a mask to obtain the third feature value. The third feature value is then used to balance the first feature value of the string by multiplying the probability to obtain the fourth feature value. Next, the fourth feature value is used as the input to the Transformer module in the character correction network module. The output of the last layer of the Transformer module is residually connected with the first feature value to obtain the final feature of each character. Finally, the final feature of each character is input into the softmax classifier, which outputs the character with the highest probability from the candidate word list as the correct character.

3. The image-text recognition and restoration method based on semantic analysis according to claim 1, characterized in that, The steps for dividing the table into cells are as follows: Within the defined table area, the super-resolution algorithm TSR is used to detect the horizontal and vertical lines in the table and calculate the intersection points of the lines to form individual cells, thus determining the structure of the table and the coordinates of each cell. Based on a tree model, continuous learning is performed using CNN as the foundation. At each decision step, a Sequential Attention model is used to select useful features for inference and a TableNet model is constructed. The TableNet model is then used to analyze the cell coordinates in the table. The final table structure is obtained by combining the coordinates of each cell.

4. The image-text recognition and restoration method based on semantic analysis according to claim 1, characterized in that, The steps for reconstructing the text in the cell to obtain the second character recognition result include: For the same text box, calculate the coordinates of the four sides of the text box and the difference between them and the cell boundary. If the difference exceeds the threshold, the text box is considered to span cells. Then, determine whether the text spanning cells is split in the direction of top, bottom, left, or right. Find the cells in the corresponding split direction, divide the text box into multiple parts using the cell boundary coordinates, and reconstruct the coordinates of the text box. For each text box in the same cell, a language model is used to calculate the combination that scores the highest for forming a sentence from the text in each text box, and finally the text in the cell is output.

5. A semantic analysis-based image and text recognition and restoration system, characterized in that, The image-text recognition and restoration method based on semantic analysis, applied to any one of claims 1-4, includes a module for obtaining an image of the text to be recognized, a text box detection model module, a first character recognition result module, and a second character recognition result module; The module for obtaining the text image to be recognized is used to obtain a first text image to be recognized and process the first text image to obtain a second text image. The text box detection model module is used to obtain the text box of the second text image by passing it through a pre-established text box detection model; The first character recognition result module is used to obtain the first character recognition result by passing the text box of the second text image through a pre-built character recognition model; The character recognition model includes a patch embedding module, a feature mixing module, a merging module, an error character localization module, and a character correction network module; the patch embedding module is used to convert a two-dimensional image into a series of one-dimensional patch blocks; The feature mixing module is used to extract character feature sequences; the merging module is used to generate transcribed sequences from the character feature sequences; the error character localization module is used to determine the location information of error characters; the character correction network module is used to correct transcribed character sequences; wherein, the feature mixing module further includes a global mixing module and a local mixing module; the global mixing module is used to evaluate the dependencies between all character components and establish dependencies between different character components; the local mixing module is used to simulate stroke example features; The second character recognition result module is used to sort the coordinates of the text boxes of the second text image in ascending order, and to detect whether there is a table in the second text image. If there is a table, the position of the table is marked. Merging text at the same height outside the table involves calculating the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference is within a threshold range, the two text boxes can be connected. The text merging process outside the table at different heights involves calculating the difference between the right coordinate of the text box and the left coordinate of the next text box. If the difference exceeds a threshold, the process calculates the difference between the right coordinate of the text box and the global right coordinate of the second text image, and the difference between the left coordinate of the next text box and the global left coordinate of the second text image. If both differences are less than the threshold, the characters in the two text boxes are merged, and the dictionary, part-of-speech logic judgment, and N-gram language model are added. Finally, the probability that the combined text of the two text boxes forms a sentence is calculated. If the probability is higher than the threshold, the text in the two text boxes is directly merged. Within the defined location range of the table, the table is divided into cells, and the text in the cells is reorganized to obtain the second character recognition result.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the semantic analysis-based image recognition and restoration method as described in any one of claims 1-4.

7. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the image and text recognition and repair method based on semantic analysis as described in any one of claims 1-4.

Images