Text image super-resolution reconstruction method and related equipment

Abstract

The embodiment of the present application belongs to the field of artificial intelligence technology and is applied to the field of smart medical care. It relates to a method for super-resolution reconstruction of text images and related equipment. A low-resolution image is input into a scene text recognition model to obtain text position and text content information; a text mask is generated based on the text position information and text content information, and the text mask is upsampled to obtain a target mask; the low-resolution image and the target mask are input into an adversarial network to obtain a discrimination result, and the discrimination accuracy is calculated based on the discrimination result; the loss function is calculated based on the low-resolution image and the target mask until the loss function converges and the discrimination accuracy is lower than the accuracy threshold, and a trained adversarial network is obtained; the received low-resolution image to be converted is input into the trained adversarial network to obtain a target super-resolution image. The trained adversarial network can be stored in a blockchain. The present application realizes the quality of super-resolution reconstruction of text images.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text image super-resolution reconstruction method and related equipment. Background Art

[0002] Super-resolution reconstruction refers to the process of generating a corresponding high-resolution image through a convolutional neural network for any given low-resolution image, preserving and restoring as much detail and texture as possible in the image. Super-resolution reconstruction technology has played a good role in promoting the development of related fields such as image classification, segmentation, tracking and defogging, and plays an important role in the development of neural networks.

[0003] However, text images are different from natural scenes. Text content has fixed shapes and clear edges, so reconstruction requirements are higher. For ordinary images, most of the scenes in the image are natural and random, so it is easier to convert low-resolution images into high-resolution ones. For text in the scene, if there is distortion, color mutation, or the text edge merges with other scenes in the reconstructed image, it will significantly reduce the quality of the reconstructed image. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a text image super-resolution reconstruction method and related equipment to ensure the quality of super-resolution reconstruction of text images.

[0005] In order to solve the above technical problems, the present application embodiment provides a text image super-resolution reconstruction method, which adopts the following technical solution:

[0006] A text image super-resolution reconstruction method comprises the following steps:

[0007] Receiving a low-resolution image and a corresponding high-resolution image, inputting the low-resolution image into a pre-trained scene text recognition model, and obtaining output text position information and text content information;

[0008] Generate a text mask based on the text position information and the text content information, and upsample the text mask to obtain a target mask;

[0009] Inputting the low-resolution image and the target mask into a generation layer of a preset adversarial network to obtain an output super-resolution image;

[0010] Inputting the super-resolution image and the high-resolution image into the discrimination layer of the adversarial network at the same time, obtaining an output discrimination result, and calculating the discrimination accuracy based on the discrimination result;

[0011] Calculating the loss function of the adversarial network based on the low-resolution image and the target mask until the loss function converges and the discrimination accuracy is lower than the accuracy threshold, thereby obtaining a trained adversarial network;

[0012] A low-resolution image to be converted is received, and the low-resolution image to be converted is input into a trained adversarial network to obtain an output target super-resolution image.

[0013] Furthermore, the step of generating a text mask based on the text position information and the text content information includes:

[0014] Correct the text position information based on the text content information to obtain target text position information;

[0015] The text mask is generated based on the target text position information.

[0016] Furthermore, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0017] The content loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the content loss function are:

[0018] in, is the content loss function, is the value of the pixel at the (x, y) position of the high-resolution image, G θG (I LR ) x，y is the value of the pixel of the super-resolution image at the (x, y) position, rW and rH are the width and length of the super-resolution image respectively, r 2 WH is the total number of pixels in the super-resolution image.

[0019] Furthermore, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0020] The adversarial loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the adversarial loss function are:

[0021] in, is the adversarial loss function, G θG (I LR ) is the super-resolution image, D θD is the discriminant layer, M is the total number of the super-resolution images, and m represents the number of the super-resolution images.

[0022] Furthermore, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0023] The regularized loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the regularized loss function are:

[0024] in, is the regularized loss function, G θG (I LR ) x，y is the value of the pixel at the (x, y) position of the super-resolution image, rW and rH are the width and length of the target mask respectively, r 2 WH is the total number of pixels in the target mask, |||| represents the norm, Represents the gradient.

[0025] Furthermore, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0026] The text-aware loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the text-aware loss function are:

[0027] Among them, l TR is the text perception loss function, N is the total number of pixels at the location where the text exists, is the target mask, is the super-resolution image.

[0028] Furthermore, the step of upsampling the text mask to obtain a target mask includes:

[0029] The text mask is upsampled multiple times to obtain the target mask.

[0030] In order to solve the above technical problems, the embodiment of the present application further provides a text image super-resolution reconstruction device, which adopts the following technical solution:

[0031] A text image super-resolution reconstruction device, comprising:

[0032] A receiving module, used to receive a low-resolution image and a corresponding high-resolution image, input the low-resolution image into a pre-trained scene text recognition model, and obtain output text position information and text content information;

[0033] An upsampling module, used for generating a text mask based on the text position information and the text content information, and upsampling the text mask to obtain a target mask;

[0034] A generation module, used for inputting the low-resolution image and the target mask into a generation layer of a preset adversarial network to obtain an output super-resolution image;

[0035] A discrimination module, used for simultaneously inputting the super-resolution image and the high-resolution image into the discrimination layer of the adversarial network, obtaining an output discrimination result, and calculating a discrimination accuracy rate based on the discrimination result;

[0036] A calculation module, used for calculating the loss function of the adversarial network based on the low-resolution image and the target mask, until the loss function converges and the discrimination accuracy is lower than the accuracy threshold, to obtain a trained adversarial network;

[0037] The acquisition module is used to receive the low-resolution image to be converted, input the low-resolution image to be converted into the trained adversarial network, and obtain the output target super-resolution image.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0039] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned text image super-resolution reconstruction method when executing the computer-readable instructions.

[0040] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0041] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the above-mentioned text image super-resolution reconstruction method.

[0042] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0043] This application obtains the text position and text content information of the received low-resolution image, and generates a text mask based on the text position information and text content information. The generation of the text mask takes into account the text position information and content information, so that the boundary between the text in the image and the surrounding image can be clearly defined, so that the text in the subsequently generated super-resolution image is clear, and the quality of the reconstructed image is significantly improved. By upsampling the text mask, the text mask is enlarged, and the resolution of the text mask is improved, so as to facilitate the subsequent generation of super-resolution images. Through the adversarial training of the generation layer and the discrimination layer in the adversarial network, the trained adversarial network is obtained, which is used to generate a target super-resolution image with better quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0046] Figure 2 is a flowchart of an embodiment of a text image super-resolution reconstruction method according to the present application;

[0047] Figure 3 is a structural schematic diagram of an embodiment of a text image super-resolution reconstruction device according to the present application;

[0048] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application.

[0049] Attached figure numbers: 200, computer device; 201, memory; 202, processor; 203, network interface; 300, text image super-resolution reconstruction device; 301, receiving module; 302, upsampling module; 303, generating module; 304, distinguishing module; 305, calculating module; 306, obtaining module. DETAILED DESCRIPTION

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0054] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0055] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0056] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0057] It should be noted that the text image super-resolution reconstruction method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the text image super-resolution reconstruction device is generally arranged in the server / terminal device.

[0058] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0059] Continue to refer Figure 2 , shows a flow chart of an embodiment of a text image super-resolution reconstruction method according to the present application. The text image super-resolution reconstruction method comprises the following steps:

[0060] S1: receiving a low-resolution image and a corresponding high-resolution image, inputting the low-resolution image into a pre-trained scene text recognition model, and obtaining output text position information and text content information.

[0061] In this embodiment, the size of the low resolution image (LRimage) is W*H. The low resolution image is input into the scene text recognition model to obtain the scene text position and content. The scene text recognition model of the present application is: the text recognition model ASTER (《Aster: An attentional scene text recognizer with flexible rectification》. The present application pre-trains the text recognition model.

[0062] In this embodiment, the text image super-resolution reconstruction method is executed on an electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the low-resolution image and the corresponding high-resolution image through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0063] S2: Generate a text mask based on the text position information and the text content information, and upsample the text mask to obtain a target mask.

[0064] In this embodiment, a text mask (textmask) that only emphasizes the text part is generated based on the text position information and the text content information, and the size of the text mask is the same as the low-resolution image. In the text mask, the pixels where the text exists are marked as 1, and the pixels where the text does not exist are marked as 0, that is, a two-dimensional mask is obtained. When the text mask size is H*W, it is a text mask of a low-resolution image. The text mask is upsampled (upsampling) to obtain a new text mask, that is, a target mask, whose size is rW*rH. At this time, the target mask size is the same as the generated high-resolution image. For the received high-resolution image, the image needs to be resized to be consistent with the size of the target mask to facilitate subsequent calculations. The target mask is used to subsequently supervise the generation results of the generation layer (i.e., super-resolution images). This application does not need to annotate high-resolution images to complete the scene text recognition part operation.

[0065] Specifically, the step of generating a text mask based on the text position information and the text content information includes:

[0066] Correct the text position information based on the text content information to obtain target text position information;

[0067] The text mask is generated based on the target text position information.

[0068] In this embodiment, a text mask is generated based on the text position information and the text content information. In the picture, some places are not clearly photographed, or the position recognition result is wrong. The computer recognizes the text content information and can generate a mask with higher accuracy. For example, if the content is not considered, the output mask may be "goed", while the network that considers the content will output "good".

[0069] In addition, as another embodiment of the present application, the step of upsampling the text mask to obtain a target mask includes:

[0070] The text mask is upsampled multiple times to obtain the target mask.

[0071] In this embodiment, the present application performs 5-fold upsampling on the text mask, enlarges the text mask by 5 times, improves the resolution of the text mask, and thus generates a super-resolution image that is 5 times larger than the low-resolution image.

[0072] S3: Input the low-resolution image and the target mask into a generation layer of a preset adversarial network to obtain an output super-resolution image.

[0073] In this embodiment, after scene text recognition, the computer generates a super-resolution image through the generation layer of the Generative Adversarial Networks (GAN): the low-resolution image and the target mask are simultaneously input into the generation layer (Generator) of the Generative Adversarial Network, and the generation layer (Generator) generates a super-resolution image (Super Resolution image, SR image).

[0074] S4: Inputting the super-resolution image and the high-resolution image into the discrimination layer of the adversarial network at the same time, obtaining an output discrimination result, and calculating the discrimination accuracy based on the discrimination result.

[0075] In this embodiment, the super-resolution image and the high-resolution image (High Resolution image, HR image) are simultaneously input into the discriminator layer (Discriminator), and the discriminator layer outputs the discrimination result, that is, outputs the super-resolution image or outputs the high-resolution image. For example, the discriminator layer outputs 0 or 1, where 0 represents "the image is a generated image (super-resolution image)", and 1 represents that the image is a real image (high-resolution image). This application uses adversarial training of the generation layer and the discriminator layer. As the super-resolution image generated by the generation layer becomes more and more similar to the high-resolution image in the natural scene, it becomes more and more difficult to distinguish. When the accuracy of the discriminator layer output is lower than the accuracy threshold, it is determined that the discriminator layer has difficulty in distinguishing whether the input is a real image or a super-resolution image generated by the generation layer, indicating that the super-resolution image generated by the generation layer is of high quality and similar to the real image, that is, the training goal is achieved and used in practical applications. Among them, the accuracy is calculated as the ratio of the number of correct discrimination results output by the discriminator layer within a preset time period to the total number of discriminations.

[0076] S5: Calculate the loss function of the adversarial network based on the low-resolution image and the target mask until the loss function converges and the discrimination accuracy is lower than the accuracy threshold, thereby obtaining a trained adversarial network.

[0077] In this embodiment, the loss function involved in this application is mainly the loss function involved in generating images by generating adversarial networks, including content loss function, adversarial loss function and regularization loss function, as well as text perceptual loss designed for text masking. When the loss function and the discrimination accuracy are lower than the accuracy threshold, it is determined that the adversarial network training is completed, and an adversarial network with better performance is obtained.

[0078] Specifically, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0079] The content loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the content loss function are:

[0080] in, is the content loss function, is the value of the pixel at the (x, y) position of the high-resolution image, G θG (I LR ) x，yis the value of the pixel of the super-resolution image at the (x, y) position, rW and rH are the width and length of the super-resolution image respectively, r 2 WH is the total number of pixels in the super-resolution image.

[0081] In this embodiment, the content loss function calculates the mean square error, and the width and length of the super-resolution image and the high-resolution image are rW and rH respectively. This application calculates the sum of the difference between the pixels at all positions of the super-resolution image and the high-resolution image with a width of rW and a length of rH, divided by the number of pixels, as the text perception loss. The loss between the super-resolution image and the high-resolution image is calculated.

[0082] As another embodiment of the present application, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0083] The adversarial loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the adversarial loss function are:

[0084] in, is the adversarial loss function, G θG (I LR ) is the super-resolution image, D θD is the discriminant layer, M is the total number of the super-resolution images, and m represents the number of the super-resolution images.

[0085] In this embodiment, the adversarial loss requires that the discriminant layer D successfully distinguish the super-resolution image generated by the generative layer G from the natural high-resolution image input therein. Through adversarial training of the generative layer and the discriminant layer, the quality of the super-resolution image generated by the network gradually improves. M is the total number of super-resolution images input to the discriminant layer.

[0086] In addition, as another embodiment of the present application, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0087] The regularized loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the regularized loss function are:

[0088] in, is the regularized loss function, G θG (I LR ) x，y is the value of the pixel at the (x, y) position of the super-resolution image, rW and rH are the width and length of the target mask respectively, r 2WH is the total number of pixels in the target mask, |||| represents the norm, Represents the gradient.

[0089] In this embodiment, a regularized loss function is added to prevent network overfitting and accelerate the convergence of the overall loss function.

[0090] As another embodiment of the present application, the step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask includes:

[0091] The text-aware loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the text-aware loss function are:

[0092] Among them, l TR is the text perception loss function, N is the total number of pixels at the location where the text exists, is the target mask, is the super-resolution image.

[0093] In this embodiment, the target mask is calculated The difference in pixel value between the position where the text exists and the corresponding position of the generated image in the generation layer, N represents the total number of pixels where the text exists. Sum all the differences and divide them by N, which is the text perception function. Through the text perception function, the generation layer will produce clearer text when constructing a new image. This application marks the pixels where the text exists in the mask as 1, and the pixels where the text does not exist as 0, and generates a target mask after upsampling. The target mask supervises the generation result of the generation layer through the text perception loss function to achieve the emphasis on text only.

[0094] S6: receiving a low-resolution image to be converted, inputting the low-resolution image to be converted into a trained adversarial network, and obtaining an output target super-resolution image.

[0095] In this embodiment, based on the trained adversarial network, a target super-resolution image with better quality can be generated to ensure that the text information in the image is clear and complete.

[0096] This application obtains the text position and text content information of the received low-resolution image, and generates a text mask based on the text position information and text content information. The generation of the text mask takes into account the text position information and content information, so that the boundary between the text in the image and the surrounding image can be clearly defined, so that the text in the subsequently generated super-resolution image is clear, and the quality of the reconstructed image is significantly improved. By upsampling the text mask, the text mask is enlarged, and the resolution of the text mask is improved, so as to facilitate the subsequent generation of super-resolution images. Through the adversarial training of the generation layer and the discrimination layer in the adversarial network, the trained adversarial network is obtained, which is used to generate a target super-resolution image with better quality.

[0097] It should be emphasized that in order to further ensure the privacy and security of the trained adversarial network, the trained adversarial network can also be stored in a node of a blockchain.

[0098] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0099] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0100] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0101] This application can be applied in the field of smart medical care to restore low-resolution images in the medical field, thereby promoting the construction of smart cities.

[0102] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0103] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0104] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a text image super-resolution reconstruction device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0105] like Figure 3As shown, the text image super-resolution reconstruction device 300 described in this embodiment includes: a receiving module 301, an upsampling module 302, a generating module 303, a discriminating module 304, a calculating module 305 and an obtaining module 306. Among them: the receiving module 301 is used to receive a low-resolution image and a corresponding high-resolution image, input the low-resolution image into a pre-trained scene text recognition model, and obtain output text position information and text content information; the upsampling module 302 is used to generate a text mask based on the text position information and the text content information, and upsample the text mask to obtain a target mask; the generating module 303 is used to input the low-resolution image and the target mask into the generating layer of a preset adversarial network to obtain an output super-resolution image; the discriminating module 304 is used to The super-resolution image and the high-resolution image are simultaneously input into the discrimination layer of the adversarial network to obtain an output discrimination result, and the discrimination accuracy is calculated based on the discrimination result; a calculation module 305 is used to calculate the loss function of the adversarial network based on the low-resolution image and the target mask until the loss function converges and the discrimination accuracy is lower than the accuracy threshold, thereby obtaining the trained adversarial network; an acquisition module 306 is used to receive the low-resolution image to be converted, input the low-resolution image to be converted into the trained adversarial network, and obtain the output target super-resolution image.

[0106] In this embodiment, the present application obtains the text position and text content information of the received low-resolution image, and generates a text mask based on the text position information and text content information. The generation of the text mask takes into account the position information and content information of the text, so that the boundary between the text in the image and the surrounding image can be clearly defined, so that the text in the subsequently generated super-resolution image is clear, and the quality of the reconstructed image is significantly improved. By upsampling the text mask, the text mask is enlarged, and the resolution of the text mask is improved, so as to facilitate the subsequent generation of super-resolution images. Through the adversarial training of the generation layer and the discrimination layer in the adversarial network, a trained adversarial network is obtained, which is used to generate a target super-resolution image with better quality.

[0107] The up-sampling module 302 includes a correction submodule and a generation submodule, wherein the correction submodule is used to correct the text position information based on the text content information to obtain the target text position information; the generation submodule is used to generate the text mask based on the target text position information.

[0108] In some optional implementations of this embodiment, the up-sampling module 302 is further used to: perform multiple up-sampling on the text mask to obtain the target mask.

[0109] In some optional implementations of this embodiment, the calculation module 305 is further used to calculate the content loss function of the adversarial network based on the low-resolution image, and the characteristics of the content loss function are:

[0110] in, is the content loss function, is the value of the pixel at the (x, y) position of the high-resolution image, G θG (I LR ) x，y is the value of the pixel of the super-resolution image at the (x, y) position, rW and rH are the width and length of the super-resolution image respectively, r 2 WH is the total number of pixels in the super-resolution image.

[0111] In some optional implementations of this embodiment, the calculation module 305 is further used to calculate the adversarial loss function of the adversarial network based on the low-resolution image, and the characteristics of the adversarial loss function are:

[0112] in, is the adversarial loss function, G θG (I LR ) is the super-resolution image, D θD is the discriminant layer, M is the total number of the super-resolution images, and m represents the number of the super-resolution images.

[0113] In some optional implementations of this embodiment, the calculation module 305 is further used to calculate the regularized loss function of the adversarial network based on the low-resolution image, and the characteristics of the regularized loss function are:

[0114] in, is the regularized loss function, G θG (I LR ) x，y is the value of the pixel at the (x, y) position of the super-resolution image, rW and rH are the width and length of the target mask respectively, r 2 WH is the total number of pixels in the target mask, |||| represents the norm, Represents the gradient.

[0115] In some optional implementations of this embodiment, the calculation module 305 is further used to calculate the text perception loss function of the adversarial network based on the low-resolution image, and the characteristics of the text perception loss function are:

[0116] Among them, l TR is the text perception loss function, N is the total number of pixels at the location where the text exists, is the target mask, is the super-resolution image.

[0117] This application obtains the text position and text content information of the received low-resolution image, and generates a text mask based on the text position information and text content information. The generation of the text mask takes into account the text position information and content information, so that the boundary between the text in the image and the surrounding image can be clearly defined, so that the text in the subsequently generated super-resolution image is clear, and the quality of the reconstructed image is significantly improved. By upsampling the text mask, the text mask is enlarged, and the resolution of the text mask is improved, so as to facilitate the subsequent generation of super-resolution images. Through the adversarial training of the generation layer and the discrimination layer in the adversarial network, the trained adversarial network is obtained, which is used to generate a target super-resolution image with better quality.

[0118] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0119] The computer device 200 includes a memory 201, a processor 202, and a network interface 203 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 200 having components 201-203, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0120] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0121] The memory 201 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 201 can be an internal storage unit of the computer device 200, such as a hard disk or memory of the computer device 200. In other embodiments, the memory 201 can also be an external storage device of the computer device 200, such as a plug-in hard disk equipped on the computer device 200, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 201 can also include both the internal storage unit of the computer device 200 and its external storage device. In this embodiment, the memory 201 is generally used to store the operating system and various application software installed on the computer device 200, such as computer-readable instructions of the text image super-resolution reconstruction method. In addition, the memory 201 can also be used to temporarily store various data that have been output or are to be output.

[0122] The processor 202 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 202 is generally used to control the overall operation of the computer device 200. In this embodiment, the processor 202 is used to run the computer-readable instructions stored in the memory 201 or process data, such as computer-readable instructions for running the text image super-resolution reconstruction method.

[0123] The network interface 203 may include a wireless network interface or a wired network interface. The network interface 203 is generally used to establish a communication connection between the computer device 200 and other electronic devices.

[0124] In this embodiment, the generation of the text mask takes into account the location information and content information of the text, thereby being able to clearly define the boundary between the text in the image and the surrounding image, significantly improving the quality of the reconstructed image. Through adversarial training of the generative layer and the discriminative layer in the adversarial network, a trained adversarial network is obtained, which is used to generate a target super-resolution image with better quality.

[0125] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the text image super-resolution reconstruction method as described above.

[0126] In this embodiment, the generation of the text mask takes into account the location information and content information of the text, thereby being able to clearly define the boundary between the text in the image and the surrounding image, significantly improving the quality of the reconstructed image. Through adversarial training of the generative layer and the discriminative layer in the adversarial network, a trained adversarial network is obtained, which is used to generate a target super-resolution image with better quality.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0128] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A text image super-resolution reconstruction method, characterized in that: The steps include: Receiving a low-resolution image and a corresponding high-resolution image, inputting the low-resolution image into a pre-trained scene text recognition model, and obtaining output text position information and text content information; Generate a text mask based on the text position information and the text content information, and upsample the text mask to obtain a target mask, wherein after obtaining the target mask, adjust the size of the high-resolution image to be consistent with the size of the target mask; Inputting the low-resolution image and the target mask into a generation layer of a preset adversarial network to obtain an output super-resolution image; Inputting the super-resolution image and the high-resolution image into the discrimination layer of the adversarial network at the same time, obtaining an output discrimination result, and calculating the discrimination accuracy based on the discrimination result; Calculating a loss function of the adversarial network based on the low-resolution image and the target mask until the loss function converges and the discrimination accuracy is lower than an accuracy threshold, thereby obtaining a trained adversarial network; A low-resolution image to be converted is received, and the low-resolution image to be converted is input into a trained adversarial network to obtain an output target super-resolution image.

2. The text image super-resolution reconstruction method according to claim 1, characterized in that: The step of generating a text mask based on the text position information and the text content information comprises: Correct the text position information based on the text content information to obtain target text position information; The text mask is generated based on the target text position information.

3. The text image super-resolution reconstruction method according to claim 1, characterized in that: The step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask comprises: The content loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the content loss function are: ,in, is the content loss function, is the value of the pixel at the (x, y) position of the high-resolution image, For the low-resolution image In the generation layer The value of the pixel at the (x, y) position of the super-resolution image generated in , rW and rH are the width and length of the super-resolution image respectively, is the total number of pixels in the super-resolution image.

4. The text image super-resolution reconstruction method according to claim 1, characterized in that: The step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask comprises: The adversarial loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the adversarial loss function are: ,in, is the adversarial loss function, For the low-resolution image In the generation layer The super-resolution image generated in is the discriminant layer, M is the total number of the super-resolution images, and m represents the number of the super-resolution images.

5. The text image super-resolution reconstruction method according to claim 1, characterized in that: The step of calculating the loss function of the adversarial network based on the low-resolution image and the target mask comprises: The regularized loss function of the adversarial network is calculated based on the low-resolution image, and the characteristics of the regularized loss function are: ,in, is the regularized loss function, is the value of the pixel at the (x, y) position of the super-resolution image, rW and rH are the width and length of the target mask respectively, is the total number of pixels in the target mask, ‖‖ represents the norm, Represents the gradient.

6. The text image super-resolution reconstruction method according to claim 1, characterized in that: The step of upsampling the text mask to obtain a target mask comprises: The text mask is upsampled multiple times to obtain the target mask.

7. A text image super-resolution reconstruction device, characterized in that: include: A receiving module, used to receive a low-resolution image and a corresponding high-resolution image, input the low-resolution image into a pre-trained scene text recognition model, and obtain output text position information and text content information; an upsampling module, configured to generate a text mask based on the text position information and the text content information, and upsample the text mask to obtain a target mask, wherein after obtaining the target mask, the size of the high-resolution image is adjusted to be consistent with the size of the target mask; A generation module, used for inputting the low-resolution image and the target mask into a generation layer of a preset adversarial network to obtain an output super-resolution image; A discrimination module, used to input the super-resolution image and the high-resolution image into the discrimination layer of the adversarial network at the same time, obtain an output discrimination result, and calculate a discrimination accuracy rate based on the discrimination result; A calculation module, used for calculating the loss function of the adversarial network based on the low-resolution image and the target mask, until the loss function converges and the discrimination accuracy is lower than the accuracy threshold, to obtain a trained adversarial network; The acquisition module is used to receive the low-resolution image to be converted, input the low-resolution image to be converted into the trained adversarial network, and obtain the output target super-resolution image.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text image super-resolution reconstruction method according to any one of claims 1 to 6 when executing the computer-readable instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text image super-resolution reconstruction method according to any one of claims 1 to 6 are implemented.

Images