Text line image deblurring method, device and computer readable medium

Abstract

The embodiment of the application provides a text line image deblurring method, device and computer readable medium. After the first text line image is processed, the text vector information, the text position information and the third feature map of each character in the first text line image can be extracted. Since the third feature map contains information about the font style of each character, the text vector information reflects the character content corresponding to each character. Therefore, in the subsequent modulation convolution processing, style transfer processing, feature map splicing and deconvolution processing, the information of the character content, the text position and the font style is used. Compared with the prior art, more effective information is used, and the deblurring effect is further improved, so that the characters in the generated second text line image are clearer.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a text line image deblurring method, device and computer readable medium. BACKGROUND

[0002] The text line image refers to an image containing a line of text. In actual scenarios, the text content in the text line image may be blurred due to various reasons. In order to better recognize the text content, the text line image needs to be deblurred to make the text content clearer. In the existing deblurring scheme, only the image information of the text line image itself is simply used to realize deblurring, and the effect is not ideal. SUMMARY

[0003] An object of the present application is to provide a text line image deblurring method, device and computer readable medium to solve the problem of poor deblurring effect in the existing scheme.

[0004] To achieve the above object, the embodiments of the present application provide a text line image deblurring method, which comprises:

[0005] Convolving the first text line image to obtain a first feature map;

[0006] Using a multi-head attention mechanism to process the first feature map, adding weight information to the feature elements in the first feature map to obtain a second feature map;

[0007] Optimizing the features of the second feature map to obtain a third feature map;

[0008] Recognizing the text content of the second feature map to obtain the text vector information of each character contained in the second feature map;

[0009] Recognizing the text position of the second feature map to obtain the text position information of each character contained in the second feature map;

[0010] Modulating and convolving according to the text vector information and the third feature map to obtain a first text feature map corresponding to each character;

[0011] According to the text position information, a second text feature map of each character is intercepted from the first feature map;

[0012] The style of the font style contained in the first text feature map of each character is migrated to the second text feature map of the corresponding character to obtain a third text feature map of each character;

[0013] According to the text position information, the third text feature maps of the respective characters are spliced to obtain a fourth feature map.

[0014] The fourth feature map is deconvolved to obtain the second text line image.

[0015] Further, the first text line image is convolved to obtain the first feature map, including:

[0016] The first text line image is convolved using the ResNet model to obtain the first feature map.

[0017] Furthermore, a multi-head attention mechanism is used to process the first feature map, adding weight information to the feature elements of the first feature map to obtain a second feature map, including:

[0018] The first feature map is processed through a multi-head attention mechanism to obtain the first processing result;

[0019] The first processing result is summed with the first feature map, and then subjected to layer normalization to obtain the second processing result.

[0020] The second processing result is processed by the first MLP model to obtain the third processing result;

[0021] The third processing result is summed with the second processing result, and after layer normalization, weight information is added to the feature elements of the first feature map to obtain the second feature map.

[0022] Further, feature optimization is performed on the second feature map to obtain a third feature map, including:

[0023] The second feature map is processed by the second MLP model to obtain the fifth processing result;

[0024] The fifth processing result is subjected to convolution operation to obtain the third feature map.

[0025] Further, text content recognition is performed on the second feature map to obtain the text vector information of each character contained in the second feature map, including:

[0026] The second feature map is processed by the third MLP model to obtain the sixth processing result;

[0027] The sixth processing result is processed by the first activation function to obtain the text vector information of each character contained in the second feature map.

[0028] Further, the sixth processing result is processed by the first activation function to obtain the text vector information of each character contained in the second feature map, including:

[0029] The sixth processing result is processed by the first activation function to obtain the index number corresponding to each character contained in the second feature map;

[0030] The text vector information corresponding to the index number is queried in the preset encoding dictionary, and the text vector information of each character contained in the second feature map is determined based on the query result.

[0031] Further, text location recognition is performed on the second feature map to obtain the text location information of each character contained in the second feature map, including:

[0032] The second feature map is processed by the fourth MLP model to obtain the seventh processing result;

[0033] The result of the seventh processing is processed by the second activation function to obtain the text position information of each character contained in the second feature map.

[0034] Further, based on the text location information, a second text feature map is extracted from the first feature map for each character, including:

[0035] The first feature map is convolved using the UNet model to obtain the fifth feature map.

[0036] Based on the text location information, the fifth feature map is processed by ROI Align to extract the second text feature map of each character;

[0037] The fourth feature map is obtained by concatenating the third text feature maps of each character based on the text location information, including:

[0038] Based on the text location information, reverse ROI Align processing is performed on the third text feature maps of each character to obtain a fourth feature map obtained by splicing the third text feature maps of each character.

[0039] Furthermore, the font style contained in the first text feature map of each character is transferred to the second text feature map of the corresponding character to obtain the third text feature map of each character, including:

[0040] The AdaIN algorithm is used to transfer the font style contained in the first text feature map of each character to the second text feature map of the corresponding character, thus obtaining the third text feature map of each character.

[0041] Some embodiments of this application also provide a text line image deblurring device, wherein the device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned text line image deblurring method.

[0042] Other embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the text line image deblurring method described above.

[0043] Compared to existing technologies, the text line image deblurring scheme provided in this application first processes the first text line image to extract the text vector information, text position information, and third feature map of each character in the first text line image. Since the third feature map contains information about the font style of each character, and the text vector information reflects the text content corresponding to each character, the subsequent modulation convolution processing, style transfer processing, feature map stitching, and deconvolution processing use information from the three aspects of text content, text position, and font style. Compared to existing schemes, this scheme utilizes more effective information, further improving the deblurring effect and making the text in the generated second text line image clearer. Attached Figure Description

[0044] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 A flowchart illustrating a text line image deblurring method provided in this application embodiment;

[0046] Figure 2 This is a flowchart illustrating the processing of the transformer layer in the embodiments of this application;

[0047] Figure 3 This is a flowchart illustrating the optimization process of the second feature map in this embodiment of the application;

[0048] Figure 4 This is a flowchart illustrating the text content recognition process performed on the second feature map in an embodiment of this application.

[0049] Figure 5 This is a flowchart illustrating the text location recognition process performed on the second feature map in an embodiment of this application.

[0050] Figure 6 A flowchart illustrating the deblurring process for text line images using the solution of this application embodiment;

[0051] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0052] The present application will now be described in further detail with reference to the accompanying drawings.

[0053] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0054] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0055] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program devices, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0056] This application provides a method for deblurring text line images. The method first processes a first text line image to extract text vector information, text position information, and a third feature map for each character. Since the third feature map contains information about the font style of each character, and the text vector information reflects the text content corresponding to each character, subsequent modulation convolution processing, style transfer processing, feature map stitching, and deconvolution processing utilize information from these three aspects: text content, text position, and font style. Compared to existing solutions, this method utilizes more effective information, further improving the deblurring effect and making the text in the generated second text line image clearer.

[0057] In practical scenarios, the execution subject of this method can be a user device, a network device, or a device composed of user devices and network devices integrated through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, and tablets; the network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud consists of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0058] Figure 1 The present application illustrates a processing flow of a text line image deblurring method according to an embodiment of this application, which includes at least the following processing steps:

[0059] Step S101: Perform convolution processing on the first text line image to obtain the first feature map. The first text line image is the image to be processed in this solution. For example, in real-world scenarios, the image resolution may be low, or the photographed object may be old, damaged, or faded, resulting in insufficient clarity of the text line image, making it difficult to distinguish the text contained within. In this case, such a text line image can be used as the first text line image in this solution, and deblurring processing can be performed on it to obtain a text line image with higher clarity.

[0060] In some embodiments of this application, convolutional layers of a ResNet (Residual Network) model can be used to extract feature maps from the first text line image. Specifically, the first text line image (Blurry text image) is processed by convolutional layering using a ResNet model to obtain the first feature map (ResNet feature map). In this embodiment, the ResNet used can be ResNet 45, which contains 45 convolutional and fully connected layers. In practical scenarios, other depths of ResNet models, such as ResNet 50 and ResNet 101, can be used as needed.

[0061] Step S102: Process the first feature map using a multi-head attention mechanism, add weight information to the feature elements in the first feature map, and obtain the second feature map.

[0062] Multi-head attention allows the first feature map to acquire weighted information, resulting in a second feature map with further weighted information. For example, elements related to the text portion have higher weights, while elements unrelated to the text portion have relatively lower weights. In this embodiment, a transformer layer can be constructed to process the first feature map and obtain the second feature map. The processing procedure of this transformer layer is as follows: Figure 2 As shown, it includes the following steps:

[0063] Step S201: The first feature map is processed by a multi-head attention mechanism to obtain a first processing result. Taking the first feature map (resnet feature map) obtained by the ResNet model as an example, Q vector, K vector, and V vector of the multi-head attention mechanism are constructed based on the resnet feature map, and after being input into the multi-head attention module for calculation, the first processing result can be obtained.

[0064] Step S202: After adding the first processing result to the first feature map (ResNet feature map), the result is then subjected to layer normalization to obtain the second processing result.

[0065] Step S203: The second processing result is processed by the first MLP (Multi-Layer Perceptron) model to obtain the third processing result.

[0066] Step S204 involves summing the third processing result with the second processing result and then performing layer normalization. This adds weight information to the feature elements of the first feature map, thereby obtaining the second feature map, the ResNet feature map. In practical scenarios, the transformer layer processing undergoes N iterations, resulting in better weight information in the second feature map, thus improving the accuracy of subsequent processing.

[0067] Step S103: Perform feature optimization on the second feature map to obtain the third feature map.

[0068] In this embodiment, the specific optimization process may include: processing the second feature map through a second MLP model to obtain a fifth processing result, and then performing a convolution operation on the fifth processing result to obtain the third feature map. Through the above optimization process, more representative feature information about font style can be extracted from the second feature map.

[0069] In practical scenarios, the optimization process can also utilize a transformer layer for N iterations. The transformer layer used in the optimization process can employ different parameters to better facilitate feature extraction of font styles. Therefore, the optimization process in this embodiment can be as follows: Figure 3 As shown, the second feature map (ResNet feature map) is first iteratively processed through N transformer layers. The processed second feature map (ResNet feature map) is then input into the second MLP model for further processing. The fifth processing result is then obtained, and the fifth processing result is then convolved to obtain the third feature map.

[0070] Step S104: Perform text content recognition on the second feature map to obtain the text vector information of each character contained in the second feature map.

[0071] In this embodiment, the specific text content recognition process may include: processing the second feature map through a third MLP model to obtain a sixth processing result, and then processing the sixth processing result through a first activation function to obtain the text vector information of each character contained in the second feature map.

[0072] In the text content recognition process, a transformer layer can be used for N iterations of optimization. The transformer layer used in the text content recognition process can also employ different parameters to better suit the text content recognition and processing. Therefore, the text content recognition process in this embodiment can be as follows: Figure 4 As shown, the second feature map ResNet feature map ' is first iteratively processed through N transformer layers. The processed second feature map ResNet feature map ' is then input into the third MLP model for further processing. The sixth processing result is then obtained, and the sixth processing result is processed through the first activation function to obtain the text vector information of each character contained in the second feature map.

[0073] In the above processing, the sixth processing result, after being processed by the first activation function, outputs an index (char index) corresponding to a preset encoding dictionary. The first activation function can be the softmax function. When the first text line image contains m characters, a total of m corresponding indexes (char indexes) can be obtained. After obtaining the indexes, vector selection (select feature) processing can be performed from the encoding dictionary (code book) using the indexes. That is, by using the indexes, the text vector information corresponding to the indexes is queried in the preset encoding dictionary, thereby determining the discrete feature of the text vector information of each character contained in the second feature map based on the query results.

[0074] In this embodiment, if the code book contains 65536 elements, each element is a vector, representing the 65536 characters in the character set, corresponding to index numbers 0 to 65535. For example, vector f1 corresponds to index number 0, vector f2 corresponds to index number 1, and so on. The code book stores the vector corresponding to each character. When a first text line image contains 5 characters, 5 char indices can be output. When the char indices of the 5 characters are 0, 300, 23, 26, and 1, it can be represented as [0, 300, 23, 26, 1]. Based on the obtained char indices, the corresponding vector is queried in the code book to find the vector corresponding to each character, thus obtaining the text vector information. Taking the aforementioned 5 characters as an example, their corresponding text vector information, discrete feature, can be represented as [f1, f301, f24, f27, f2].

[0075] Step S105: Perform text position recognition on the second feature map to obtain the text position information of each character contained in the second feature map.

[0076] In this embodiment, the specific text location recognition process may include: processing the second feature map through a fourth MLP model to obtain a seventh processing result, and then processing the seventh processing result through a second activation function to obtain the text location information of each character contained in the second feature map.

[0077] In the text location recognition process, a transformer layer can be used for N iterations of optimization. The transformer layer used in the text location recognition process can also employ different parameters to better suit the text location recognition processing. Therefore, the text location recognition process in this embodiment can be as follows:Figure 5 As shown, the second feature map ResNet feature map is first iteratively processed through N transformer layers. The processed second feature map ResNet feature map is then input into the fourth MLP model for further processing. The seventh processing result is then obtained, and the seventh processing result is processed through the second activation function to obtain the text position information (char box) of each character contained in the second feature map.

[0078] The text location information can be the coordinate information of each character in the second feature map. The coordinate information corresponding to each character can be represented in the form of [x-coordinate of the top left corner, y-coordinate of the top left corner, x-coordinate of the bottom right corner, and y-coordinate of the bottom right corner], with the specific position of each character determined by the diagonal x and y coordinates. For example, if a first text line image contains 5 characters, this step can obtain 5 sets of text location information, namely char box1 to char box5. Furthermore, the second activation function used in this embodiment can be a ReLU function.

[0079] The first to fourth MLP models used in the above steps can be designed with appropriate structures and parameters according to the processing tasks they perform, so as to better complete the corresponding processing tasks and obtain more accurate processing results.

[0080] Step S106: Perform modulated convolution processing based on the text vector information and the third feature map to obtain the first text feature map corresponding to each character. The text vector information mainly contains feature information about the text content, while the third feature map mainly contains feature information about the font style. By performing modulated convolution processing on both, a first text feature map (proir char feature map) that can be used for the font style features of each character is extracted from the third feature map. For example, for the scenario of 5 characters in the aforementioned embodiment, 5 first text feature maps can be extracted, namely proir char feature maps 1 to 5. Furthermore, in practical scenarios, to extract more accurate first text feature maps, N modulated convolution processes can be performed to obtain the first text feature map corresponding to each character.

[0081] Step S107: Based on the text location information, extract the second text feature map of each character from the first feature map.

[0082] In some embodiments of this application, the first feature map can be convolved using the UNet model to obtain the fifth feature map. Taking the aforementioned first feature map, the ResNet feature map, as an example, after convolving it with the convolutional layers of the UNet model, the fifth feature map, the Unet feature map, can be obtained. Then, based on the text position information, the fifth feature map is processed by ROI Align to extract the second text feature map for each character. For example, if the image contains 5 characters, and their corresponding text position information is char box1 to char box5, then based on this text position information, the fifth feature map, the Unet feature map, can be processed by ROI Align (Region of Interest Align) to extract the second text feature maps char feature maps1 to 5 for these 5 characters.

[0083] Step S108 involves transferring the font style contained in the first text feature map of each character to the second text feature map of the corresponding character, thereby obtaining the third text feature map of each character. The style transfer process can be performed using the AdaIN (Adaptive Instance Normalization) algorithm. In this embodiment, by employing the AdaIN algorithm, the font style contained in the first text feature map (proir char feature map) of each character can be transferred to the second text feature map (char feature map) of the corresponding character, resulting in the third text feature map (adain char feature map) of each character. For example, in the scenario of the aforementioned five characters, the third text feature maps adain char feature maps 1~5 for the five characters can be obtained.

[0084] The AdaIN algorithm can transfer the font style features contained in the first text feature map to the second text feature map of each character, so that the third text feature map obtained in this way also carries the corresponding font style features.

[0085] Step S109: Based on the text location information, the third text feature maps of each character are concatenated to obtain the fourth feature map.

[0086] In this embodiment, based on the text location information, reverse ROI alignment processing can be performed on the third text feature maps of each character to obtain a fourth feature map obtained by concatenating the third text feature maps of each character. For example, for 5 characters, reverse ROI alignment processing can be performed on the adana char feature maps 1-5 of these 5 characters based on char boxes 1-5 to obtain a fourth feature map featuremap' concatenated from the 5 adana char feature maps. Since the text location information carries positional features of the characters in the image, the fourth feature map feature map' obtained after concatenation based on the text location information will also carry corresponding positional features. The third text feature map itself carries features about font style and text content. Therefore, the fourth feature map obtained simultaneously contains feature information such as font style, text content, and text position, which is beneficial for subsequently reconstructing a clearer text line image.

[0087] Step S110: Perform deconvolution (transposed conv) processing on the fourth feature map to obtain the second text line image. The second text line image (HD text image) is the output image after deblurring. Since this scheme first processes the first text line image, it can extract the text vector information, text position information, and third feature map of each character in the first text line image. Because the third feature map contains information about the font style of each character, and the text vector information reflects the text content corresponding to each character, the subsequent modulation convolution processing, style transfer processing, feature map stitching, and deconvolution processing utilize information from these three aspects: text content, text position, and font style. Compared to existing schemes, this utilizes more effective information, further improving the deblurring effect and making the text in the generated second text line image clearer.

[0088] Figure 6 The present invention illustrates a process for deblurring text line images using an embodiment of this application, comprising at least the following processing steps:

[0089] Step S601: Using the ResNet45 model, perform preliminary feature map extraction on the first text line image (Blurry text image) to obtain the first feature map (ResNet feature map).

[0090] Step S602: Iteratively execute N transformer layer processes on the ResNet feature map. The specific steps of each transformer layer process are as follows: After processing the ResNet feature map through multi-head attention, add the first processing result to the ResNet feature map using an add operation. Then, perform layer normalization to obtain the second processing result. Next, process the second processing result through the first MLP model to obtain the third processing result. Add the second processing result output by layer normalization to the third processing result output by the first MLP model using an add operation. After further layer normalization, obtain the second feature map, ResNet feature map'.

[0091] Step S603, execute the font style prediction process: the output ResNet feature map from step S602 is processed through N transformer layers, then processed by the second MLP model to obtain the fifth processing result. The fifth processing result is then subjected to convolution operation, and the output is the third feature map of the entire text line image.

[0092] Step S604: Perform the character classification process: The output ResNet feature map from step S602 is processed through N transformer layers, then through the third MLP model to obtain the sixth processing result. This sixth processing result is then processed by the softmax function, and the output is the index number (char index) corresponding to each character in the first text line image. If the first text line image contains m characters, there are also m index numbers, i.e., char index 1 to char index m.

[0093] Step S605: Execute the character bounding box process: The output ResNet feature map from step S602 is processed through N transformer layers, then through the fourth MLP model, resulting in the seventh processing result. This seventh processing result is then processed by the ReLU function, and the output is the text position information for each character in the first text line image. The text position information can be expressed as the x and y coordinates of the top-left and bottom-right corners, resulting in m character boxes, numbered from character box 1 to character box m.

[0094] Step S606: Execute the prior generation process: Based on the index number obtained in step S604, query the code book for the corresponding text vector, obtaining the text vectors discretefeatures corresponding to m characters. The code book contains 65536 elements, each with an index number from 0 to 65535, representing 65536 characters. The value of each element is a text vector discrete feature. Then, the discretefeatures and the feature map obtained in step S603 are used together as input for N iterations of modulated convolution (conv) processing, yielding the first text feature map (proir char feature map 1 ~ proir char feature map m) corresponding to each character.

[0095] Step S607: Input the ResNet feature map into the UNet model and output the fifth feature map, the UNET feature map.

[0096] In step S608, the ROI Align operation is performed on the unet feature map and the m groups of char boxes obtained in step S605, and the unet feature map is truncated into m char feature maps, namely char feature map 1 to char feature map m.

[0097] Step S609: The m proir char feature maps and m char feature maps obtained in step S606 are processed using the AdaIN algorithm. The font style contained in the proir char feature map of each character is transferred to the corresponding char feature map to obtain m third text feature maps adain char feature maps, namely adain char feature map 1 to adain char feature map m.

[0098] In step S610, the m adain char feature maps and the m groups of char boxes obtained in step S605 are combined to perform a reverse ROI alignment operation to obtain the fourth feature map ' composed of the m adain char feature maps.

[0099] Step S611: The feature map' is processed by N deconvolutions (transposed conv) to obtain a high-resolution second text image (HD text image).

[0100] Based on another aspect of this application, embodiments of this application also provide a text line image deblurring device, the device including a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned text line image deblurring method.

[0101] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0102] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0103] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0104] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the various embodiments of this application.

[0107] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0108] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any particular order. Numbers corresponding to steps are used to label and distinguish different steps, and the magnitude of these numbers does not limit any particular order of execution.

Claims

1. A method for deblurring text lines in an image, characterized in that, The method includes: Perform convolution processing on the first text line image to obtain the first feature map; The first feature map is processed through a multi-head attention mechanism to obtain the first processing result; The first processing result is summed with the first feature map, and then subjected to layer normalization to obtain the second processing result. The second processing result is processed by the first MLP model to obtain the third processing result; The third processing result is summed with the second processing result, and after layer normalization, weight information is added to the feature elements of the first feature map to obtain the second feature map. The second feature map is optimized to obtain the third feature map; Text content recognition is performed on the second feature map to obtain the text vector information of each character contained in the second feature map; Perform text location recognition on the second feature map to obtain the text location information of each character contained in the second feature map; Modulation convolution processing is performed based on the text vector information and the third feature map to obtain the first text feature map corresponding to each character; Based on the text location information, extract a second text feature map for each character from the first feature map; The font style contained in the first text feature map of each character is transferred to the second text feature map of the corresponding character to obtain the third text feature map of each character; The fourth feature map is obtained by concatenating the third text feature maps of each character based on the text location information. The fourth feature map is deconvolved to obtain the second text line image.

2. The method according to claim 1, characterized in that, The first text line image is convolved to obtain the first feature map, including: The first text line image is convolved using the ResNet model to obtain the first feature map.

3. The method according to claim 1, characterized in that, The second feature map is optimized to obtain the third feature map, including: The second feature map is processed by the second MLP model to obtain the fifth processing result; The fifth processing result is subjected to convolution operation to obtain the third feature map.

4. The method according to claim 1, characterized in that, Perform text content recognition on the second feature map to obtain the text vector information of each character contained in the second feature map, including: The second feature map is processed by the third MLP model to obtain the sixth processing result; The sixth processing result is processed by the first activation function to obtain the text vector information of each character contained in the second feature map.

5. The method according to claim 4, characterized in that, The sixth processing result is processed by the first activation function to obtain the text vector information of each character contained in the second feature map, including: The sixth processing result is processed by the first activation function to obtain the index number corresponding to each character contained in the second feature map; The text vector information corresponding to the index number is queried in the preset encoding dictionary, and the text vector information of each character contained in the second feature map is determined based on the query result.

6. The method according to claim 1, characterized in that, Perform text location recognition on the second feature map to obtain the text location information of each character contained in the second feature map, including: The second feature map is processed by the fourth MLP model to obtain the seventh processing result; The result of the seventh processing is processed by the second activation function to obtain the text position information of each character contained in the second feature map.

7. The method according to claim 1, characterized in that, Based on the text location information, a second text feature map is extracted from the first feature map for each character, including: The first feature map is convolved using the UNet model to obtain the fifth feature map. Based on the text location information, the fifth feature map is processed by ROI Align to extract the second text feature map of each character; The fourth feature map is obtained by concatenating the third text feature maps of each character based on the text location information, including: Based on the text location information, reverse ROI Align processing is performed on the third text feature maps of each character to obtain a fourth feature map obtained by splicing the third text feature maps of each character.

8. The method according to claim 1, characterized in that, The font style contained in the first text feature map of each character is transferred to the second text feature map of the corresponding character to obtain the third text feature map of each character, including: The AdaIN algorithm is used to transfer the font style contained in the first text feature map of each character to the second text feature map of the corresponding character, thus obtaining the third text feature map of each character.

9. A text line image deblurring device, wherein, The device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method of any one of claims 1 to 8.

10. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 8.

Images