Image processing method and related apparatus thereof

Abstract

The embodiment of the application provides an image processing method and related equipment, relates to the field of artificial intelligence, and particularly relates to the field of computer vision. In the method, strokes in the original image are detected by a first stroke detection network to obtain a second mask indicating the positions of the strokes in the original image, and then a first stroke elimination network eliminates the strokes in the original image according to the second mask to obtain a first target image, that is, the first target image is obtained by eliminating the strokes in the original image with the mask indicating the positions of the strokes in the original image as the granularity of the stroke elimination, compared with the stroke elimination process with the rectangular mask in which the strokes are located as the granularity, the method avoids the loss of image information in the background of the strokes in the rectangular mask, thereby increasing the continuity of the image information in the image after the stroke elimination to some extent and improving the effect of rendering and backfilling in the stroke elimination process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image processing method and related equipment. Background Technology

[0002] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Computer vision is an important research area within AI, specifically referring to machine vision that uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. Image inpainting (or image completion), as an important research area in computer vision, aims to reasonably fill in missing parts of an image to obtain a visually complete image. Generally, in existing common image inpainting tasks, the region to be repaired is defined by a binary rectangular mask with the same resolution as the image to be processed.

[0004] Currently, text removal is one of the many application scenarios for image restoration. The processing of text removal tasks follows the common image restoration technology, that is, in the area of ​​the image to be restored, using the rectangular mask where the text is located as the granularity, the fill content that is consistent with the surrounding background of the mask in terms of texture and structure is used as the content for rendering backfill in the mask, so that in the image after text removal, the fill content is consistent with the image context outside the mask, thus realizing text removal.

[0005] However, the area to be repaired corresponding to the rectangular mask containing the text often contains not only the text itself but also the text background. The text removal process described above is based on the rectangular mask containing the text, which can easily lead to the loss of image information in the text background. Since the image information in the text background cannot be restored, the image information in the image after text removal is not coherent, resulting in poor text removal effect. Summary of the Invention

[0006] This application provides an image processing method and related equipment to increase the coherence of image information in an image after text removal and improve the rendering backfilling effect during the text removal process.

[0007] The first aspect of this application provides an image processing method. In this method, a first mask corresponding to an original image is determined based on the original image. The first mask is used to indicate the position of a target character in the original image. The original image includes the target character, which is composed of character strokes. Then, the original image and the first mask are input into a first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the character strokes in the original image. The first stroke detection network is used to detect character strokes in the image. Further, at least the original image and the second mask are input into a first stroke removal network. After processing by the first stroke removal network, a first target image is obtained. The first stroke removal network is used to remove character strokes from the image. The first target image is the image after removing character strokes from the original image. The first stroke detection network is used to detect text strokes in the image, and the first stroke removal network is used to remove text strokes in the image. That is, after the first stroke detection network detects the text strokes in the original image and obtains a second mask indicating the position of the text stroke in the original image, the first stroke removal network performs text removal in the original image according to the second mask to obtain the first target image. That is, the first target image is obtained by performing text removal in the original image with the mask indicating the position of the text stroke as the granularity of text removal. Compared with the text removal process implemented with the rectangular mask where the text is located as the granularity, it avoids the loss of image information in the text background of the rectangular mask, thereby increasing the coherence of image information in the image after text removal to a certain extent and improving the rendering backfill effect during the text removal process.

[0008] In one possible implementation of the first aspect of this application, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module. Specifically, the process of inputting the original image and the first mask into the first stroke detection network and obtaining the second mask after processing by the first stroke detection network includes: inputting the original image and the first mask into the first stroke detection network; then, the first downsampling module downsampling the original image according to the first mask to obtain a first feature of the original image; further, the first upsampling module upsampling the first feature of the original image to obtain the second mask.

[0009] In this embodiment, the first stroke detection network used to detect text strokes in an image can specifically be a U-shaped network. This U-shaped network includes a first downsampling module and a first upsampling module. Specifically, the process of the first stroke detection network obtaining the second mask involves the first downsampling module downsampling and extracting the first features of the original image, and then using the first upsampling module to upsample the first features of the original image to obtain the second mask. This provides a specific implementation method for the first stroke detection network to obtain the second mask, improving the feasibility of this solution and thus increasing its implementation flexibility.

[0010] In one possible implementation of the first aspect of this application, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module; the first stroke detection network further includes a fifth processing module and / or a sixth processing module, wherein the fifth processing module is used to increase the model depth of the first stroke detection network, and the sixth processing module is used to increase the number of model parameters of the first stroke detection network; inputting the original image and the first mask into the first stroke detection network, and obtaining the second mask after processing by the first stroke detection network specifically includes: inputting the original image and the first mask into the first stroke detection network; the first downsampling module downsampling the original image according to the first mask to obtain a fourth feature of the original image; the fourth feature of the original image being processed by the fifth processing module and / or the sixth processing module to obtain a fifth feature of the original image; and the first upsampling module upsampling the fifth feature of the original image to obtain the second mask.

[0011] In this embodiment, the first stroke detection network may further include a fifth processing module for increasing the model depth of the first stroke detection network. For example, the fifth processing module may be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the first stroke detection network may further include a sixth processing module for increasing the number of model parameters of the first stroke detection network. For example, the second processing module may be used to perform convolution processing, dilated convolution processing, residual block processing, etc., thereby improving the text detection effect of the first stroke detection network in detecting character strokes through the fifth processing module and / or the sixth processing module, and improving the stroke detection effect in the text detection process.

[0012] In one possible implementation of the first aspect of this application, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module. Specifically, the process of inputting at least the original image and the second mask into the first stroke elimination network to obtain the first target image after processing by the first stroke elimination network may include: inputting at least the original image and the second mask into the first stroke elimination network; then, the second downsampling module downsamples the original image according to the second mask to obtain a second feature of the original image; subsequently, the second upsampling module upsamples the second feature of the original image to obtain the first target image.

[0013] In this embodiment, the first stroke elimination network used to remove text strokes in an image can specifically be a U-shaped network. This U-shaped network includes a second downsampling module and a second upsampling module. Specifically, the process of the first stroke elimination network obtaining the second mask involves the second downsampling module downsampling the original image to extract its second features, and then using the second upsampling module to upsample the second features of the original image to obtain the first target image. This provides a specific implementation method for the first stroke elimination network to obtain the first target image, improving the feasibility of this solution and thus increasing its implementation flexibility.

[0014] In one possible implementation of the first aspect of this application, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; the first stroke elimination network further includes a first processing module and / or a second processing module, wherein the first processing module is used to increase the model depth of the first stroke elimination network, and the second processing module is used to increase the number of model parameters of the first stroke elimination network; at least the original image and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network specifically includes: at least the original image and the second mask are input into the first stroke elimination network; the second downsampling module downsamples the original image according to the second mask to obtain a second feature of the original image; the second feature of the original image is processed by the first processing module and / or the second processing module to obtain a third feature of the original image; the second upsampling module upsamples the third feature of the original image to obtain the first target image.

[0015] In this embodiment, the first stroke elimination network may further include a first processing module for increasing the model depth of the first stroke elimination network. For example, the first processing module may be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the first stroke elimination network may further include a second processing module for increasing the number of model parameters of the first stroke elimination network. For example, the second processing module may be used to perform convolution processing, dilated convolution processing, residual block processing, etc., thereby improving the text elimination effect of the first stroke elimination network when eliminating text strokes through the first processing module and / or the second processing module, and further improving the rendering backfill effect during the text elimination process.

[0016] In one possible implementation of the first aspect of the present application, the process of inputting at least the original image and the second mask into the first stroke elimination network and obtaining the first target image after processing by the first stroke elimination network may specifically include: inputting the original image, the first mask and the second mask into the first stroke elimination network and obtaining the first target image after processing by the first stroke elimination network.

[0017] In this embodiment, the original image, the first mask, and the second mask are input into the first stroke elimination network, so that the first stroke elimination network performs text elimination at the location of the first mask in the original image according to the second mask. Compared with the process of performing text elimination according to the second mask in the entire original image, more attention can be paid to the area where the target text indicated by the first mask is located in the original image, thereby further improving the text elimination effect.

[0018] In one possible implementation of the first aspect of this application, after at least the original image and the second mask are input into a first stroke elimination network and processed by the first stroke elimination network to obtain a first target image, the method may further include: inputting at least the first target image and the second mask into a second stroke detection network and processing by the second stroke detection network to obtain a third mask, wherein the second stroke detection network is used to detect text strokes in the image; thereafter, at least the first target image and the third mask are input into a second stroke elimination network and processed by the second stroke elimination network to obtain a second target image, wherein the second stroke elimination network is used to eliminate text strokes in the image.

[0019] In this embodiment, after the first target image is obtained by processing the original image through the first stroke detection network and the first stroke elimination network, the first target image can be further processed through the second stroke detection network and the second stroke elimination network to obtain the second target image. Here, the combination of a character stroke detection network and a character stroke elimination network is called an algorithm unit. In this embodiment, data flow between the two algorithm units can further improve the character elimination effect.

[0020] In one possible implementation of the first aspect of this application, the second stroke detection network can be a U-shaped network, including a third downsampling module and a third upsampling module; wherein, at least the first target image and the second mask are input into the second stroke detection network, and the third mask is obtained after processing by the second stroke detection network, specifically including: at least the first target image and the second mask are input into the second stroke detection network; then, the third downsampling module downsamples the first target image according to the second mask to obtain a first feature of the first target image; thereafter, the third upsampling module upsamples the first feature of the first target image to obtain the third mask.

[0021] In this embodiment, the second stroke detection network used to detect text strokes in an image can specifically be a U-shaped network. This U-shaped network includes a third downsampling module and a third upsampling module. Specifically, the process of the second stroke detection network obtaining the third mask involves the third downsampling module downsampling and extracting the first feature of the first target image, followed by the third upsampling module upsampling the first feature of the first target image to obtain the third mask. This provides a specific implementation method for the second stroke detection network to obtain the third mask, improving the feasibility and flexibility of this solution.

[0022] In one possible implementation of the first aspect of this application, the second stroke detection network is a U-shaped network, including a third downsampling module and a third upsampling module; the second stroke detection network further includes a seventh processing module and / or an eighth processing module, wherein the seventh processing module is used to increase the model depth of the second stroke detection network, and the eighth processing module is used to increase the number of model parameters of the second stroke detection network; at least the first target image and the second mask are input into the second stroke detection network, and the third mask is obtained after processing by the second stroke detection network specifically includes: at least the first target image and the second mask are input into the second stroke detection network; the third downsampling module downsamples the original image according to the second mask to obtain the sixth feature of the original image; the sixth feature of the original image is processed by the seventh processing module and / or the eighth processing module to obtain the seventh feature of the original image; the third upsampling module upsamples the seventh feature of the original image to obtain the third mask.

[0023] In this embodiment, the second stroke detection network may further include a seventh processing module for increasing the model depth of the second stroke detection network. For example, the seventh processing module may be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the second stroke detection network may further include an eighth processing module for increasing the number of model parameters of the second stroke detection network. For example, the second processing module may be used to perform convolution processing, dilated convolution processing, residual block processing, etc. Thus, the seventh processing module and / or the eighth processing module can improve the text detection effect of the second stroke detection network when detecting text strokes, and improve the stroke detection effect during the text detection process.

[0024] In one possible implementation of the first aspect of this application, the second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module. Specifically, the process of inputting at least the first target image and the third mask into the second stroke elimination network to obtain the second target image after processing by the second stroke elimination network may include: inputting at least the first target image and the third mask into the second stroke elimination network; then, the fourth downsampling module downsamples the first target image according to the third mask to obtain a second feature of the first target image; subsequently, the fourth upsampling module upsamples the second feature of the first target image to obtain the second target image.

[0025] In this embodiment, the second stroke elimination network used to remove text strokes in an image can specifically be a U-shaped network. This U-shaped network includes a fourth downsampling module and a fourth upsampling module. Specifically, the process of the second stroke elimination network obtaining the second target image involves the fourth downsampling module downsampling the first target image to extract its second features, and then using the fourth upsampling module to upsample the second features of the first target image to obtain the second target image. This provides a specific implementation method for the second stroke elimination network to obtain the second target image, improving the feasibility and flexibility of this solution.

[0026] In one possible implementation of the first aspect of this application, the second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module; the second stroke elimination network further includes a third processing module and / or a fourth processing module, wherein the third processing module is used to increase the model depth of the second stroke elimination network, and the fourth processing module is used to increase the number of model parameters of the second stroke elimination network; at least the first target image and the third mask are input into the second stroke elimination network, and the second target image is obtained after processing by the second stroke elimination network specifically includes: at least the first target image and the third mask are input into the second stroke elimination network; the fourth downsampling module downsamples the first target image according to the third mask to obtain a second feature of the first target image; the second feature of the first target image is processed by the third processing module and / or the fourth processing module to obtain a third feature of the first target image; the fourth upsampling module upsamples the third feature of the first target image to obtain the second target image.

[0027] In this embodiment, the second stroke elimination network may further include a third processing module for increasing the model depth of the second stroke elimination network. For example, the third processing module may be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the second stroke elimination network may further include a fourth processing module for increasing the number of model parameters of the second stroke elimination network. For example, the fourth processing module may be used to perform convolution processing, dilated convolution processing, residual block processing, etc., thereby improving the text elimination effect of the first stroke elimination network when eliminating text strokes through the third processing module and / or the fourth processing module, and further improving the rendering backfill effect during the text elimination process.

[0028] In one possible implementation of the first aspect of the present application, the process of inputting at least the first target image and the second mask into the second stroke detection network and obtaining the third mask after processing by the second stroke detection network may specifically include: inputting the first target image, the first mask and the second mask into the second stroke detection network and obtaining the third mask after processing by the second stroke detection network.

[0029] In this embodiment, the first target image, the first mask, and the second mask are input into the second stroke detection network. The second stroke detection network performs character stroke detection at the location of the first mask in the first target image based on the second mask. Compared with the process of performing character stroke detection based on the second mask in the entire first target image, more attention can be paid to the area where the target character indicated by the first mask is located in the first target image, improving the character stroke detection effect and thus improving the subsequent character removal effect.

[0030] In one possible implementation of the first aspect of the present application, the process of inputting the first target image and the third mask into the second stroke elimination network and obtaining the second target image after processing by the second stroke elimination network may specifically include: inputting the first target image, the first mask and the third mask into the second stroke elimination network and obtaining the second target image after processing by the second stroke elimination network.

[0031] In this embodiment, the first target image, the first mask, and the third mask are input into the second stroke elimination network, so that the second stroke elimination network performs text elimination at the location of the first mask in the first target image according to the third mask. Compared with the process of performing text elimination according to the third mask in the entire first target image, more attention can be given to the area where the target text indicated by the first mask is located in the first target image, thereby further improving the text elimination effect.

[0032] In one possible implementation of the first aspect of this application, the first stroke detection network further includes a first loss function, wherein the first loss function includes:

[0033]

[0034] Where M is the first mask, M s For the second mask, M gt For M g The corresponding truth value, λ d The integer is greater than 1; the larger the value, the higher the weight of the text region. ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network.

[0035] In this embodiment, the first stroke detection network can update its training parameters using the function value of the first loss function, providing a specific implementation method for the loss function of the first stroke detection network, improving the feasibility of this solution, and thus increasing the flexibility of its implementation.

[0036] In one possible implementation of the first aspect of this application, the first stroke elimination network further includes a second loss function, wherein the second loss function includes:

[0037]

[0038] Among them, I te For the first target image, I gt For I te The corresponding truth value, λ r The integer is greater than 1; the larger the value, the higher the weight of the text region. ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the character stroke elimination network.

[0039] In this embodiment, the first stroke elimination network can update its training parameters using the function value of the second loss function, providing a specific implementation method for the loss function of the first stroke elimination network, improving the feasibility of this solution, and thus enhancing the implementation flexibility of this solution.

[0040] In one possible implementation of the first aspect of the present application, the method may further include: inputting the first target image and the ground truth image of the first target image into a first weighted discriminator, and obtaining a first evaluation result after processing by the first weighted discriminator, wherein the first evaluation result is used to indicate the difference between the first target image and the ground truth image of the first target image.

[0041] In this embodiment, after the first stroke elimination network processes and obtains the first target image, the first target image and its ground truth image can be input into the first weighted discriminator. After processing by the first weighted discriminator, a first evaluation result is obtained. Subsequently, the first stroke elimination network can update its training parameters through the first evaluation result, which can further improve the text elimination effect of the first stroke elimination network.

[0042] In one possible implementation of the first aspect of the embodiments of this application, the first weighted discriminator includes:

[0043] First identification formula:

[0044]

[0045] Second identification formula:

[0046]

[0047] Where, label ijk This refers to the value of the (i,j,k)th element in the patch, where ω1 is the weight of that element, and n×n×C is the size of the feature block (patch).

[0048] In this embodiment, the first weighted discriminator can be specifically implemented through the first discrimination formula and the second discrimination formula. In this embodiment, the authenticity of the original image with a larger proportion of character strokes has a greater impact on the final result of the first weighted discriminator. Thus, a specific implementation method for the first weighted discriminator is provided, which improves the feasibility of this solution and thus improves the implementation flexibility of this solution.

[0049] In one possible implementation of the first aspect of the embodiments of this application, the method further includes:

[0050] The second target image and its ground truth image are input into a second weighted discriminator. After processing by the second weighted discriminator, a second evaluation result is obtained. The second evaluation result is used to indicate the difference between the second target image and its ground truth image.

[0051] In this embodiment, after the second stroke elimination network processes the second target image, the second target image and its ground truth image can be input into the second weighted discriminator. After processing by the second weighted discriminator, a second evaluation result is obtained. Subsequently, the second stroke elimination network can update its training parameters through the second evaluation result, which can further improve the text elimination effect of the second stroke elimination network.

[0052] In one possible implementation of the first aspect of this application, the second weighted discriminator includes:

[0053] Third identification formula:

[0054]

[0055] Fourth identification formula:

[0056]

[0057] Where, label ijk This refers to the value of the (i,j,k)th element in the patch, where ω2 is the weight of that element, and n×n×C is the size of the feature block (patch).

[0058] In this embodiment, the second weighted discriminator can be specifically implemented through the third and fourth discrimination formulas. The authenticity of the second target image, which has a larger proportion of character strokes, has a greater impact on the final result of the first weighted discriminator. Thus, a specific implementation method for the second weighted discriminator is provided, which improves the feasibility of this solution and enhances its implementation flexibility.

[0059] A second aspect of this application provides an image processing method, including:

[0060] Step 1: Based on the original image, determine the first mask corresponding to the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, which is composed of character strokes.

[0061] Step 2: Input the original image and the first mask into the first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the character stroke in the original image. The first stroke detection network is used to detect character strokes in the image.

[0062] Step 3: Input the original image and the second mask into the first stroke removal network. After processing by the first stroke removal network, the first target image is obtained. The first stroke removal network is used to remove text strokes in the image. The first target image is the image after text removal of the original image.

[0063] Replace the first mask with the second mask, and repeat step 2 until the first preset condition is met; and / or,

[0064] Replace the original image with the first target image, and repeat step 3 until the second preset condition is met; and / or,

[0065] Replace the original image with the first target image, and repeat steps 2 and 3 until the third preset condition is met.

[0066] In this embodiment, the first stroke detection network in step 2 is used to detect text strokes in the image, and the first stroke elimination network in step 3 is used to eliminate text strokes in the image. That is, after detecting the text strokes in the original image through the first stroke detection network in step 2 and obtaining the second mask used to indicate the position of the text strokes in the original image, the first stroke elimination network performs text elimination in the original image according to the second mask in step 3 to obtain the first target image. That is, the first target image is obtained by performing text elimination in the original image with the mask used to indicate the position of the text strokes in the original image as the granularity of text elimination. Compared with the text elimination process implemented with the rectangular mask where the text is located as the granularity, it avoids the loss of image information in the text background in the rectangular mask. That is, it increases the coherence of image information in the image after text elimination to a certain extent and improves the rendering backfilling effect in the text elimination process. In addition, repeating steps 2 and / or 3 until the preset conditions (first preset condition, second preset condition and / or third preset condition) are met can improve the stroke detection effect and stroke elimination effect to a certain extent through the repeated execution of the first stroke detection network and the second stroke elimination network, and further improve the rendering backfill effect during the text elimination process.

[0067] It should be noted that the preset conditions (first preset condition, second preset condition and / or third preset condition) can be pre-configured or determined in response to user input commands, such as the number of executions, execution duration, execution cycle, or other parameters, which are not limited here.

[0068] In one possible implementation of the second aspect of this application, in step 2, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module; wherein, the specific process of inputting the original image and the first mask into the first stroke detection network in step 2, and obtaining the second mask after processing by the first stroke detection network, may specifically include: inputting the original image and the first mask into the first stroke detection network; then, the first downsampling module downsamples the original image according to the first mask to obtain a first feature of the original image; thereafter, the first upsampling module upsamples the first feature of the original image to obtain the second mask.

[0069] In one possible implementation of the second aspect of this application, in step 2, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module; the first stroke detection network further includes a fifth processing module and / or a sixth processing module, wherein the fifth processing module is used to increase the model depth of the first stroke detection network, and the sixth processing module is used to increase the number of model parameters of the first stroke detection network; inputting the original image and the first mask into the first stroke detection network, and obtaining the second mask after processing by the first stroke detection network specifically includes: inputting the original image and the first mask into the first stroke detection network; the first downsampling module downsampling the original image according to the first mask to obtain a fourth feature of the original image; the fourth feature of the original image being processed by the fifth processing module and / or the sixth processing module to obtain a fifth feature of the original image; and the first upsampling module upsampling the fifth feature of the original image to obtain the second mask.

[0070] In one possible implementation of the second aspect of this application, in step 3, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; wherein, the process of inputting at least the original image and the second mask into the first stroke elimination network in step 3, and obtaining the first target image after processing by the first stroke elimination network, specifically may include: inputting the original image and the second mask into the first stroke elimination network; then, the second downsampling module downsampling the original image according to the second mask to obtain the second feature of the original image; thereafter, the second upsampling module upsampling the second feature of the original image to obtain the first target image.

[0071] In one possible implementation of the second aspect of this application, in step 3, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; the first stroke elimination network further includes a first processing module and / or a second processing module, wherein the first processing module is used to increase the model depth of the first stroke elimination network, and the second processing module is used to increase the number of model parameters of the first stroke elimination network; at least the original image and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network specifically includes: at least the original image and the second mask are input into the first stroke elimination network; the second downsampling module downsamples the original image according to the second mask to obtain a second feature of the original image; the second feature of the original image is processed by the first processing module and / or the second processing module to obtain a third feature of the original image; the second upsampling module upsamples the third feature of the original image to obtain the first target image.

[0072] In one possible implementation of the second aspect of the present application, the process of inputting the original image and the second mask into the first stroke elimination network in step 3, and obtaining the first target image after processing by the first stroke elimination network, may specifically include: inputting the original image, the first mask and the second mask into the first stroke elimination network, and obtaining the first target image after processing by the first stroke elimination network.

[0073] In one possible implementation of the second aspect of this application, in step 2, the first stroke detection network further includes a first loss function, wherein the first loss function includes:

[0074]

[0075] Where M is the first mask, M s For the second mask, M gt For M s The corresponding truth value, λ d For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network.

[0076] In one possible implementation of the second aspect of this application, in step 3, the first stroke elimination network further includes a second loss function, wherein the second loss function includes:

[0077]

[0078] Among them, L te For the first target image, I gt For I te The corresponding truth value, λ rFor integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the character stroke elimination network.

[0079] In one possible implementation of the second aspect of the present application, the method further includes: step 4, inputting the first target image and the ground truth image of the first target image into a first weighted discriminator, and obtaining a first evaluation result after processing by the first weighted discriminator, the first evaluation result being used to indicate the difference between the first target image and the ground truth image of the first target image.

[0080] In one possible implementation of the second aspect of this application, in step 4, the first weighted discriminator includes:

[0081] First identification formula:

[0082]

[0083] Second identification formula:

[0084]

[0085] Where, label ijk This refers to the value of the (i,j,k)th element in the patch, where ω1 is the weight of that element, and n×n×C is the size of the feature block (patch).

[0086] For the specific implementation steps of the second aspect of this application and the various possible implementations of the second aspect, as well as the beneficial effects brought about by each possible implementation, please refer to the description of the various possible implementations in the first aspect, which will not be repeated here.

[0087] A third aspect of this application provides an image processing apparatus, comprising: a determining unit, configured to determine a first mask corresponding to an original image, the first mask indicating the position of a target character in the original image, the original image including the target character, the target character being composed of character strokes; a first processing unit, configured to input the original image and the first mask into a first stroke detection network, and obtain a second mask after processing by the first stroke detection network, the second mask indicating the position of the character strokes in the original image, the first stroke detection network detecting character strokes in the image; and a second processing unit, configured to input at least the original image and the second mask into a first stroke elimination network, and obtain a first target image after processing by the first stroke elimination network, the first stroke elimination network eliminating character strokes in the image, the first target image being an image after character elimination of the original image.

[0088] The first stroke detection network is used to detect text strokes in the image, and the first stroke removal network is used to remove text strokes in the image. That is, the first processing unit detects the text strokes in the original image through the first stroke detection network and obtains a second mask that indicates the position of the text strokes in the original image. Then, the second processing unit performs text removal in the original image based on the second mask through the first stroke removal network to obtain the first target image. In other words, the first target image is obtained by performing text removal in the original image with the mask that indicates the position of the text strokes in the original image as the granularity of text removal. Compared with the text removal process implemented with the rectangular mask where the text is located as the granularity, it avoids the loss of image information in the text background of the rectangular mask, thereby increasing the coherence of image information in the image after text removal to a certain extent and improving the rendering backfill effect during the text removal process.

[0089] In the third aspect of this application, the constituent modules of the image processing apparatus can also be used to perform the steps executed in various possible implementations of the first aspect, all of which can be referred to the first aspect and will not be repeated here.

[0090] A fourth aspect of this application provides an image processing apparatus, comprising: a determining unit, configured to determine, in step 1, a first mask corresponding to an original image, the first mask indicating the position of a target character in the original image, the original image including the target character, the target character being composed of character strokes; a first processing unit, configured to input the original image and the first mask into a first stroke detection network in step 2, and obtain a second mask after processing by the first stroke detection network, the second mask indicating the position of the character strokes in the original image, the first stroke detection network being used to detect character strokes in the image; and a second processing unit, configured to, in step 3, at least input the original image and the first mask into a first stroke detection network, and obtain a second mask after processing by the first stroke detection network, the second mask indicating the position of the character strokes in the original image, the first stroke detection network being used to detect character strokes in the image; and a second processing unit, configured to, in step 3, input at least .... The second mask is input to the first stroke elimination network. After processing by the first stroke elimination network, a first target image is obtained. The first stroke elimination network is used to eliminate text strokes in the image. The first target image is the image after text elimination of the original image. The first processing unit is also used to replace the first mask with the second mask and repeat step 2 until the first preset condition is met. And / or, the second processing unit is also used to replace the original image with the first target image and repeat step 3 until the second preset condition is met. And / or, the first processing unit and the second processing unit are also used to replace the original image with the first target image and repeat steps 2 and 3 until the third preset condition is met.

[0091] In this embodiment, the first stroke detection network in step 2 of the first processing unit is used to detect text strokes in the image, and the first stroke elimination network in step 3 of the second processing unit is used to eliminate text strokes in the image. That is, after the first processing unit detects the text strokes in the original image through the first stroke detection network in step 2 and obtains a second mask indicating the position of the text strokes in the original image, the first stroke elimination network in step 3 of the second processing unit performs text elimination in the original image according to the second mask to obtain the first target image. That is, the first target image is obtained by performing text elimination in the original image with the mask indicating the position of the text strokes in the original image as the granularity of text elimination. Compared with the text elimination process implemented with the rectangular mask where the text is located as the granularity, the loss of image information in the text background of the rectangular mask is avoided. That is, by doing so, the continuity of image information in the image after text elimination is increased to a certain extent, and the rendering backfilling effect in the text elimination process is improved. In addition, repeating steps 2 and / or 3 until the preset conditions (first preset condition, second preset condition and / or third preset condition) are met can improve the stroke detection effect and stroke elimination effect to a certain extent through the repeated execution of the first stroke detection network and the first stroke elimination network, and further improve the rendering backfill effect during the text elimination process.

[0092] It should be noted that the preset conditions (first preset condition, second preset condition and / or third preset condition) can be pre-configured or determined in response to user input commands, such as the number of executions, execution duration, execution cycle, or other parameters, which are not limited here.

[0093] In the fourth aspect of this application, the constituent modules of the image processing apparatus can also be used to perform the steps executed in various possible implementations of the second aspect, all of which can be referred to the second aspect and will not be repeated here.

[0094] Fifthly, embodiments of this application provide an image processing apparatus, including a processor and a memory coupled together. The memory stores program instructions, which, when executed by the processor, cause the apparatus to implement the image processing method described in the first or second aspect. The apparatus may be an electronic device (such as a terminal device or a server device); or it may be a component of an electronic device, such as a chip.

[0095] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method described in the first or second aspect above.

[0096] In a seventh aspect, embodiments of this application provide a circuit system including a processing circuit configured to perform the image processing method described in the first or second aspect above.

[0097] Eighthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the image processing method described in the first or second aspect above.

[0098] Ninthly, this application provides a chip system including a processor for supporting an image processing device in implementing the functions involved in the first or second aspect described above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the image processing device or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0099] The technical effects of aspects five through nine or any of their possible implementations can be found in the first aspect or the technical effects of different possible implementations of the first aspect, or in the second aspect or the technical effects of different possible implementations of the second aspect, and will not be repeated here.

[0100] As can be seen from the above technical solutions, in some embodiments provided by this application, a first mask corresponding to the original image is determined based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, which is composed of character strokes. Then, the original image and the first mask are input into a first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the character strokes in the original image. The first stroke detection network is used to detect character strokes in the image. Further, the original image and the second mask are input into a first stroke elimination network. After processing by the first stroke elimination network, a first target image is obtained. The first stroke elimination network is used to eliminate character strokes in the image. The first target image is the image after character elimination of the original image. The first stroke detection network is used to detect text strokes in the image, and the first stroke removal network is used to remove text strokes in the image. That is, after the first stroke detection network detects the text strokes in the original image and obtains a second mask indicating the position of the text stroke in the original image, the first stroke removal network performs text removal in the original image according to the second mask to obtain the first target image. That is, the first target image is obtained by performing text removal in the original image with the mask indicating the position of the text stroke as the granularity of text removal. Compared with the text removal process implemented with the rectangular mask where the text is located as the granularity, it avoids the loss of image information in the text background of the rectangular mask, thereby increasing the coherence of image information in the image after text removal to a certain extent and improving the rendering backfill effect during the text removal process. Attached Figure Description

[0101] Figure 1 A schematic diagram of the main framework of artificial intelligence provided in the embodiments of this application;

[0102] Figure 2-1 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0103] Figure 2-2 A schematic diagram of another structure of the convolutional neural network provided in the embodiments of this application;

[0104] Figure 3 A schematic diagram of the structure of a chip provided in an embodiment of this application;

[0105] Figure 4 A schematic diagram of an image processing method provided in an embodiment of this application;

[0106] Figure 5-1 This is another schematic diagram of an image processing method provided in an embodiment of this application;

[0107] Figure 5-2 This is another schematic diagram of an image processing method provided in an embodiment of this application;

[0108] Figure 5-3 This is another schematic diagram of an image processing method provided in an embodiment of this application;

[0109] Figure 6 This is another schematic diagram of an image processing method provided in an embodiment of this application;

[0110] Figure 7 This is another schematic diagram of an image processing method provided in an embodiment of this application;

[0111] Figure 8 This is another schematic diagram of an image processing method provided in an embodiment of this application;

[0112] Figure 9 A schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0113] Figure 10 This is another schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0114] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0115] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0116] To facilitate understanding, the application scenarios involved in this application will first be described exemplarily below. Specifically, through... Figure 1 This application introduces an example of the artificial intelligence framework used, through... Figure 2-1 , Figure 2-2 This application introduces an example of the artificial neural network used in this application, through... Figure 3 This paper presents an example of the chip hardware implementation used in this application.

[0117] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0118] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0119] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0120] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0121] (1) Infrastructure:

[0122] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc.

[0123] For example, sensors communicate with the outside world to acquire data, which is then provided to intelligent chips in a distributed computing system provided by the underlying platform for computation.

[0124] (2) Data

[0125] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0126] (3) Data processing

[0127] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0128] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0129] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0130] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0131] (4) General ability

[0132] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0133] (5) Smart Products and Industry Applications

[0134] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0135] As a specific application of artificial intelligence systems, Convolutional Neural Networks (CNNs) are deep neural networks with convolutional structures. They are a type of deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNNs are feed-forward artificial neural networks where each neuron responds to overlapping regions in the input image.

[0136] like Figure 2-1 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0137] Convolutional / pooling layers 120:

[0138] Convolutional layers:

[0139] like Figure 2-1 The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0140] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image.

[0141] Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0142] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0143] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0144] Pooling layer:

[0145] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 2-1Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0146] Neural network layer 130:

[0147] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 2-1 As shown in layers 131, 132 to 13n) and output layer 140, the parameters contained in these multiple hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0148] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2-1 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2-1 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0149] It should be noted that, as Figure 2-1The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 2-2 The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0150] Figure 3 This is a chip hardware structure diagram provided in an embodiment of the present invention. Figure 2-1 and Figure 2-2 The algorithm based on convolutional neural networks shown can be used in... Figure 3 This is implemented in the NPU chip shown. The neural network processor NPU 30 is mounted as a coprocessor on the host CPU, which allocates tasks to it. The core of the NPU is the arithmetic circuit 30, which, through the controller 304, controls the arithmetic circuit 303 to retrieve matrix data from memory and perform multiplication operations.

[0151] In some implementations, the arithmetic circuit 303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0152] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 308.

[0153] Unified memory 306 is used to store input and output data. Weight data is directly accessed via the Direct Memory Access Controller (DMAC) 305 and transferred to the weight memory 302. Input data is also transferred to unified memory 306 via DMAC.

[0154] BIU stands for Bus Interface Unit, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer 309.

[0155] The Bus Interface Unit (BIU) 310 is used by the instruction fetch memory 309 to fetch instructions from external memory, and also by the memory access controller 305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0156] The DMAC is mainly used to move input data from external memory DDR to unified memory 306, or to weight data to weight memory 302, or to input data to input memory 301.

[0157] The vector computation unit has over 307 processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computation in non-convolutional / FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0158] In some implementations, the vector computation unit 307 can store the processed output vector into a unified buffer 306. For example, the vector computation unit 307 can apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0159] The instruction fetch buffer 309 connected to the controller 304 is used to store the instructions used by the controller 304;

[0160] Unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0161] in, Figure 2-1 and Figure 2-2 The operations of each layer in the convolutional neural network shown can be performed by matrix computation unit 212 or vector computation unit 307.

[0162] For example, the processing procedures involved in computer vision can be applied to the above. Figure 1 Within the main framework of artificial intelligence shown above, and can be found in the aforementioned... Figure 3 On the chip hardware shown, execution Figure 2-1 and Figure 2-2The diagram illustrates the implementation of single-layer or multi-layer convolutional neural networks. Computer vision, in this context, refers to machine vision that uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting 'information' from images or multi-dimensional data.

[0163] Image inpainting (or image completion), an important area of ​​research in computer vision, aims to fill in missing parts of an image to achieve a visually complete image. The importance of image inpainting technology has become increasingly prominent with the development of digital information technology. It has significant applications in many fields, such as the restoration and preservation of ancient artifacts, image editing, film and television special effects production, and virtual reality technology. This wide range of applications has driven the development of related research. Before the advent of deep learning, there was extensive research on digital image inpainting based on traditional methods, achieving good results. After the rise of deep learning, deep neural networks demonstrated significant advantages in image feature representation and semantic extraction, attracting researchers to apply deep learning to image inpainting. The combination of the two has indeed yielded very attractive results, especially after the introduction of the concept of generative adversarial networks (GANs). Deep learning-based image inpainting methods have achieved results surpassing traditional methods in many aspects.

[0164] Among the many applications of image restoration technology, text removal in natural scenes is a subfield with relatively little research. Its task is to replace text in a given area of ​​an image with content that is consistent with the surrounding background in terms of texture and structure.

[0165] In the process of text removal in natural scenes, scene text is a crucial component of scene information. With the development of deep learning, research on scene text has proliferated, with rapid progress in various research directions, such as text detection and text recognition. As these technologies mature, their industrial applications are becoming increasingly diverse. Many applications involve processing natural scene images, which frequently contain text. When this text involves personal information such as addresses and phone numbers, there is a risk of privacy breaches. Designing an algorithm to remove text from images can eliminate this potential risk. However, text removal in natural scenes has not yet received sufficient attention.

[0166] Currently, text removal can be seen as a specific application scenario for image restoration tasks, in which text is the area that needs to be restored.

[0167] Currently, there are few targeted text removal methods. The few existing implementations simply port general image inpainting methods to text removal tasks without considering the specific characteristics of text removal. Therefore, the results are often unsatisfactory, exhibiting problems such as inconsistencies between the removed fill content and the context (e.g., texture, boundaries), residual text, and poor performance when the text occupies a large area. Specifically, general image inpainting methods use a rectangular mask containing the text as the granularity, using fill content with the same texture and structure as the surrounding background. This fill content is then rendered within the mask, ensuring consistency between the fill content and the surrounding image context in the post-texture image, thus achieving text removal. However, the area to be repaired corresponding to the rectangular mask often contains not only the text but also its background. Since the text removal process operates on a granular basis (the rectangular mask), it easily leads to the loss of image information in this background. Because this background information cannot be restored, the image information in the post-texture image is discontinuous, resulting in poor text removal performance.

[0168] In summary, existing text removal tasks are implemented using the general image inpainting process. However, in text removal tasks, the area to be repaired is only the area covered by the text strokes. But it is very difficult to obtain the fine outline of such text strokes. This is a significant difference between text removal tasks and image inpainting tasks, namely, it is very difficult to obtain the fine target area in text removal tasks.

[0169] To address the specific characteristics of text removal tasks, this application provides an image processing method and related equipment. It designs a text removal algorithm based on character stroke detection. Firstly, this application proposes that the text removal task should be decoupled into two sub-tasks: character stroke detection and text removal. A character stroke detection network is responsible for meticulously segmenting the text from the image, while the character stroke removal network is responsible for removing the text from the image based on the segmentation results. Secondly, the discriminator of existing generative adversarial networks is optimized, enabling the discriminator to pay more attention to the text regions in the image according to the needs of the text removal task. This allows for better identification of the authenticity of the discriminator's input, thus providing stronger constraints on the generator.

[0170] Furthermore, the image processing method and related equipment proposed in this application can significantly improve the rendering backfilling effect when performing text removal on natural scenes containing text; at the same time, the image processing method and related equipment proposed in this application can also achieve multilingual text removal and selective text removal, and specific applications can include HiTouch image translation, etc.

[0171] The image processing method and related equipment provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0172] Please see Figure 4 This application provides an image processing method, including:

[0173] S101. Determine the first mask corresponding to the original image based on the original image;

[0174] In this embodiment, the smart terminal determines a first mask corresponding to the original image based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, which is composed of character strokes.

[0175] Specifically, the smart terminal can be implemented as a personal computer, computer workstation, smartphone, tablet computer, server, etc. This application does not limit the implementation of the smart terminal.

[0176] Before the execution of step S101, the smart terminal can acquire the original image through its own camera, video camera, or other image acquisition devices. Alternatively, the smart terminal can acquire the original image by receiving information from other devices through its communication port; this is not limited here. Furthermore, the original image acquired by the smart terminal includes target text, which is composed of character strokes.

[0177] During the execution of step S101, the smart terminal can determine the location information of the target text in the acquired original image, i.e., the first mask. The location information indicated by the first mask can be a location identified by a pixel coordinate system, a location identified by an image coordinate system, or a location identified by other methods, which is not limited here. The smart terminal can determine the first mask corresponding to the original image based on the location information. The first mask can indicate the position of the target text in the original image. Specifically, the position can indicate a rectangular area, an elliptical area, a polygonal convex hull area, or other areas of other shapes that indicate the location of the target text, which is not limited here.

[0178] S102. Input the original image and the first mask into the first stroke detection network, and obtain the second mask after processing by the first stroke detection network.

[0179] In this embodiment, the original image and the first mask are input into a first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the text strokes in the original image, and the first stroke detection network is used to detect the text strokes in the image. In this embodiment and subsequent embodiments, the smart terminal can obtain the first stroke detection network (and possibly a first stroke elimination network / second stroke detection network / second stroke elimination network, etc.) in various ways. For example, the first stroke detection network can be pre-stored in the smart terminal's storage module, or it can be carried in information received by the smart terminal from other devices, or it can be implemented in other ways, which are not limited here.

[0180] In step S102, the first stroke detection network can obtain the position of the character stroke in the original image, i.e., the second mask. The position information indicated by the second mask can be the position identified by the pixel coordinate system, or the position identified by the image coordinate system, or the position identified by other means, which is not limited here.

[0181] In one possible implementation, the first stroke detection network can be a U-shaped network (U-NET), which can include a first downsampling module and a first upsampling module. Specifically, during step S102, the process of the smart terminal inputting the original image and the first mask into the first stroke detection network, and obtaining the second mask after processing by the first stroke detection network, can include: the smart terminal inputting the original image and the first mask into the first stroke detection network; then, the first downsampling module in the first stroke detection network downsampling the original image according to the first mask to obtain a first feature of the original image; further, the first upsampling module in the first stroke detection network upsampling the first feature of the original image to obtain the second mask.

[0182] For example, the U-shaped network can be as follows: Figure 5-1 As shown, in this U-shaped network, the input data is processed through feature extraction and feature decoding to obtain the output data, which is then output on the output side. Taking this diagram as an example, the input data in the U-shaped network (G D In the feature extraction process, after three downsampling steps, the number of features extracted from the input data gradually increases (16 / 32 / 64 / 128). Then, after three upsampling steps, the number of features extracted is further reduced (128 / 64 / 32 / 16).

[0183] Specifically, such as Figure 5-2As shown in the network structure diagram, a network with a U-Net structure is used. First, the input original image is downsampled to extract high-level features. Then, the extracted features are decoded through upsampling to obtain the precise distribution of character strokes in the image, i.e., the second mask. During the upsampling process, skip connections are used to superimpose the feature maps obtained from the upsampling process with the corresponding downsampling feature maps to better utilize the extracted features. In step S102, this U-shaped network, with G... D If we represent the first stroke detection network, then:

[0184] M s =G D (I,M)

[0185] Where M is the first mask, I is the original image, and M is the output of the first stroke detection network. s This is called a stroke mask, or the second mask.

[0186] In one possible implementation, a loss function for optimizing the first stroke detection network can be further added. In this case, the first stroke detection network also includes a first loss function, which includes:

[0187]

[0188] Where M is the first mask, M g For the second mask, M gt For M s The corresponding truth value, λ d The integer is greater than 1; the larger the value, the higher the weight of the text region. ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network. At this point, the first stroke detection network can update its training parameters using the value of this first loss function. This provides a specific implementation method for the loss function of the first stroke detection network, improving the feasibility of this scheme and thus increasing its implementation flexibility.

[0189] In one possible implementation, the first stroke detection network for detecting character strokes in an image can be a U-shaped network. Besides including a first downsampling module and a first upsampling module, the first stroke detection network may further include a fifth processing module and / or a sixth processing module. The fifth processing module is used to increase the model depth of the first stroke detection network, and the sixth processing module is used to increase the number of model parameters of the first stroke detection network. Therefore, in step S102, inputting the original image and the first mask into the first stroke detection network, and obtaining the second mask after processing by the first stroke detection network, specifically includes: inputting the original image and the first mask into the first stroke detection network; the first downsampling module downsampling the original image according to the first mask to obtain a fourth feature of the original image; the fourth feature of the original image being processed by the fifth processing module and / or the sixth processing module to obtain a fifth feature of the original image; and the first upsampling module upsampling the fifth feature of the original image to obtain the second mask. The first stroke detection network may further include a fifth processing module for increasing the model depth of the first stroke detection network. For example, the fifth processing module may be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the first stroke detection network may further include a sixth processing module for increasing the number of model parameters of the first stroke detection network. For example, the second processing module may be used to perform convolution processing, dilated convolution processing, residual block processing, etc. Thus, the fifth processing module and / or the sixth processing module can improve the text detection effect of the first stroke detection network when detecting text strokes, and improve the stroke detection effect in the text detection process.

[0190] S103. At least the original image and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network.

[0191] In this embodiment, the smart terminal inputs at least the original image and the second mask into the first stroke elimination network. After processing by the first stroke elimination network, a first target image is obtained. The first stroke elimination network is used to eliminate text strokes in the image, and the first target image is the image after text elimination of the original image.

[0192] In one possible implementation, the first stroke elimination network is a U-NET, which includes a second downsampling module and a second upsampling module. Specifically, during step S103, the smart terminal inputs at least the original image and the second mask into the first stroke elimination network. The process of obtaining the first target image after processing by the first stroke elimination network can include: inputting at least the original image and the second mask into the first stroke elimination network; then, the second downsampling module downsamples the original image according to the second mask to obtain a second feature of the original image; subsequently, the second upsampling module upsamples the second feature of the original image to obtain the first target image. The implementation of the U-NET is similar to that described in step S102, and can be achieved through... Figure 5-1 The implementation details are omitted here.

[0193] In one possible implementation, the process of inputting at least the original image and the second mask into the first stroke elimination network in step S103 to obtain the first target image after processing by the first stroke elimination network may specifically include: inputting the original image, the first mask, and the second mask into the first stroke elimination network, and obtaining the first target image after processing by the first stroke elimination network, so that the first stroke elimination network performs text elimination based on the second mask at the location of the first mask in the original image. Compared with the process of performing text elimination based on the second mask in the entire original image, more attention can be given to the area where the target text indicated by the first mask is located in the original image, thereby further improving the text elimination effect.

[0194] In one possible implementation, the first stroke elimination network for removing text strokes in an image further includes a first processing module for increasing the model depth of the first stroke elimination network; and / or, the first stroke elimination network further includes a second processing module for increasing the number of model parameters of the first stroke elimination network. Specifically, during the execution of step S103, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; the first stroke elimination network also includes a first processing module and / or a second processing module, wherein the first processing module is used to increase the model depth of the first stroke elimination network, and the second processing module is used to increase the number of model parameters of the first stroke elimination network; at this time, at least the original image and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network, specifically including: at least the original image and the second mask are input into the first stroke elimination network; the second downsampling module downsamples the original image according to the second mask to obtain the second feature of the original image; the second feature of the original image is processed by the first processing module and / or the second processing module to obtain the third feature of the original image; the second upsampling module upsamples the third feature of the original image to obtain the first target image.

[0195] For example, the first processing module can be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the first stroke elimination network can also include a second processing module for increasing the number of model parameters of the first stroke elimination network. For example, the second processing module can be used to perform convolution processing, dilated convolution processing, residual block processing, etc., thereby improving the text elimination effect of the first stroke elimination network when eliminating text strokes through the first processing module and / or the second processing module, and further improving the rendering backfill effect during the text elimination process.

[0196] Here, as Figure 5-2 As shown in the network structure diagram, taking the second mask as the pixel-level annotation of the text stroke positions in the original image, and the original image including RGB three-channel data as an example, the first stroke elimination network can eliminate the text in the original image in step S103. Using the original image, the first mask, and the two masks as input, after downsampling for feature extraction, the first stroke elimination network can further extract features from the obtained feature map through a residual block structure. Adding residual blocks instead of ordinary convolutional layers avoids degradation when the network is deep, and also aims to enable the network to learn higher-level semantic features. Skip connections are also added during upsampling to utilize shallow features. Using G... RIndicates the elimination of the network in the first stroke, I te The output of the first stroke removal network, i.e., the first target image, can be represented by the following formula, which describes the working mechanism of the stroke removal network:

[0197] I te =G R (I,M,M s )

[0198] Where M is the first mask, I is the original image, and M is the output of the first stroke detection network. s This is called a stroke mask, or the second mask.

[0199] In one possible implementation, the first stroke elimination network further includes a second loss function, wherein the second loss function comprises:

[0200]

[0201] Among them, I te For the first target image, I gt For I te The corresponding truth value, λ r The integer is greater than 1; the larger the value, the higher the weight of the text region. ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the character stroke elimination network. Subsequently, the first stroke elimination network can update its training parameters using the value of this second loss function. This provides a concrete implementation of the loss function for the first stroke elimination network, improving the feasibility and flexibility of this solution.

[0202] In one possible implementation, after step S103, the method may further include: inputting the first target image and its ground truth image into a first weighted discriminator, and obtaining a first evaluation result after processing by the first weighted discriminator. This first evaluation result indicates the difference between the first target image and its ground truth image. Specifically, after the first stroke removal network processes the first target image, the first target image and its ground truth image can be input into the first weighted discriminator. After processing by the first weighted discriminator, the first evaluation result is obtained. Subsequently, the first stroke removal network can update its training parameters using this first evaluation result, further improving the text removal effect of the first stroke removal network.

[0203] In this application, a first weighted discriminator (weightedPatchGAN) based on feature blocks is designed to identify the authenticity of the input image block by block, resulting in a patch of shape n×n×C. Figure 6In the middle ①), each element corresponds to the result of identifying the authenticity of a specific image patch based on specific features; all elements in the patch are weighted ( Figure 6 (②) After taking the average, the final discriminator result is obtained. When weighting the elements in the patch, the weight is proportional to the proportion of text strokes in the image patch corresponding to that element. Image patches with a larger proportion of text have a larger weight in the final result. Therefore, the following formula assigns the discrimination result:

[0204]

[0205] label ijk This refers to the value of the (i,j,k)th element in the patch, where ω1 is the weight of that element. After the above assignment operation, the image patch with a larger proportion of character strokes has a greater impact on the final result of the discriminator. For a given image, the first weighted discriminator calculates its final identification result using the following formula:

[0206]

[0207] Where, label ijk This refers to the value of the (i,j,k)th element in the patch, where ω1 is the weight of that element, and n×n×C is the size of the feature block (patch).

[0208] For example, such as Figure 6 This is the structure of the first weighted discriminator based on patches proposed in this paper, which consists of two branches: a normal convolutional branch D and a mask branch D. M Branch D is the image I' with text removed. te Or the truth image I gt The input is a 5×5 kernel for each convolutional layer in this branch, with the depth shown in the figure. Except for the first convolutional layer, which has a stride of 1, all other convolutional layers have a stride of 2. The output is a feature tensor branch D with a size of 8×8×256. M Taking the mask M corresponding to the image as input, the size and stride of the convolution kernel are consistent with the kernel size in branch D, but the depth is only 1, and all elements of the convolution kernel are 1. The output of this branch is a feature tensor of size 8×8×1, where the values ​​of the elements are proportional to the proportion of the character strokes. After obtaining two outputs of the same size but different depths, the two outputs are multiplied pixel by pixel to obtain the final output of size 8×8×256.

[0209] In this embodiment, the first stroke detection network is used to detect text strokes in the image, and the first stroke elimination network is used to eliminate text strokes in the image. That is, after the first stroke detection network detects the text strokes in the original image and obtains a second mask indicating the position of the text stroke in the original image, the first stroke elimination network performs text elimination in the original image according to the second mask to obtain the first target image. That is, the first target image is obtained by performing text elimination in the original image with the mask indicating the position of the text stroke as the granularity of text elimination. Compared with the text elimination process implemented with the rectangular mask where the text is located as the granularity, it avoids the loss of image information in the text background of the rectangular mask, thereby increasing the coherence of image information in the image after text elimination to a certain extent and improving the rendering backfill effect during the text elimination process.

[0210] based on Figure 4 In the image processing described in the embodiments, if a combination of a stroke detection network and a stroke elimination network is referred to as an algorithm unit, then this application can further construct an overall network containing two algorithm units, with data flowing between the two algorithm units, thereby further improving the quality of the relevant results.

[0211] In one possible implementation, Figure 4 In the illustrated embodiment, after step S203, that is, after at least the original image and the second mask are input into the first stroke elimination network and processed by the first stroke elimination network to obtain the first target image, the method may further include: the smart terminal inputs the first target image and the second mask into the second stroke detection network, and after processing by the second stroke detection network to obtain the third mask, the second stroke detection network being used to detect text strokes in the image; thereafter, at least the first target image and the third mask are input into the second stroke elimination network again, and after processing by the second stroke elimination network to obtain the second target image, the second stroke elimination network being used to eliminate text strokes in the image.

[0212] In this embodiment, after the first target image is obtained by processing the original image through the first stroke detection network and the first stroke elimination network, the first target image can be further processed through the second stroke detection network and the second stroke elimination network to obtain the second target image. The combination of a character stroke detection network and a character stroke elimination network is called an algorithm unit. In this embodiment, data is exchanged between the two algorithm units, thereby further improving the character elimination effect.

[0213] In one possible implementation, the second stroke detection network can be a U-shaped network, including a third downsampling module and a third upsampling module. Specifically, inputting at least the first target image and the second mask into the second stroke detection network, and processing them to obtain the third mask, may include: inputting at least the first target image and the second mask into the second stroke detection network; then, the third downsampling module downsamples the first target image based on the second mask to obtain a first feature of the first target image; subsequently, the third upsampling module upsamples the first feature of the first target image to obtain the third mask. The implementation process of the third downsampling module and the third upsampling module in the second stroke detection network can be compared with the aforementioned implementation process of the first downsampling module and the first upsampling module in the first stroke detection network, the only difference being that the input data is replaced, resulting in a corresponding change in the output data. This implementation process will not be elaborated further here.

[0214] Specifically, the process of inputting at least the first target image and the second mask into the second stroke detection network to obtain the third mask after processing by the second stroke detection network can include: inputting the first target image, the first mask, and the second mask into the second stroke detection network, and obtaining the third mask after processing by the second stroke detection network. More specifically, inputting the first target image, the first mask, and the second mask into the second stroke detection network allows the network to perform character stroke detection at the location of the first mask in the first target image based on the second mask. Compared to performing character stroke detection based on the second mask throughout the entire first target image, this approach can focus more attention on the area of ​​the target text indicated by the first mask in the first target image, improving the character stroke detection effect and thus enhancing the subsequent character removal effect.

[0215] Furthermore, in the implementation of the second stroke detection network, the second stroke detection network can be a U-shaped network. In addition to including the third downsampling module and the third upsampling module, the second stroke detection network can further include a seventh processing module and / or an eighth processing module. The seventh processing module is used to increase the model depth of the second stroke detection network, and the eighth processing module is used to increase the number of model parameters of the second stroke detection network. Thus, the process of inputting at least the first target image and the second mask into the second stroke detection network and obtaining the third mask after processing by the second stroke detection network can specifically include: inputting at least the first target image and the second mask into the second stroke detection network; the third downsampling module downsampling the original image according to the second mask to obtain the sixth feature of the original image; the sixth feature of the original image is processed by the seventh processing module and / or the eighth processing module to obtain the seventh feature of the original image; the third upsampling module upsampling the seventh feature of the original image to obtain the third mask.

[0216] The second stroke detection network may further include a seventh processing module for increasing the model depth of the second stroke detection network. For example, the seventh processing module may be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or the second stroke detection network may further include an eighth processing module for increasing the number of model parameters of the second stroke detection network. For example, the second processing module may be used to perform convolution processing, dilated convolution processing, residual block processing, etc. Thus, the seventh processing module and / or the eighth processing module can improve the text detection effect of the second stroke detection network when detecting text strokes, and improve the stroke detection effect during the text detection process.

[0217] In one possible implementation, the second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module. Specifically, the process of inputting at least the first target image and the third mask into the second stroke elimination network to obtain the second target image after processing by the second stroke elimination network can include: inputting at least the first target image and the third mask into the second stroke elimination network; then, the fourth downsampling module downsamples the first target image according to the third mask to obtain a second feature of the first target image; subsequently, the fourth upsampling module upsamples the second feature of the first target image to obtain the second target image. The implementation process of the third downsampling module and the third upsampling module in the second stroke detection network can be compared with the implementation process of the first downsampling module and the first upsampling module in the aforementioned first stroke detection network, the only difference being that the input data is replaced, resulting in a corresponding change in the output data. This implementation process will not be elaborated further here.

[0218] Specifically, the process of inputting at least the first target image and the third mask into the second stroke elimination network and obtaining the second target image after processing by the second stroke elimination network can include: inputting the first target image, the first mask, and the third mask into the second stroke elimination network and obtaining the second target image after processing by the second stroke elimination network.

[0219] Specifically, the first target image, the first mask, and the third mask are input into the second stroke elimination network, so that the second stroke elimination network performs text elimination at the location of the first mask in the first target image according to the third mask. Compared with the process of performing text elimination according to the third mask in the entire first target image, more attention can be given to the area where the target text indicated by the first mask is located in the first target image, thereby further improving the text elimination effect.

[0220] Furthermore, in the second stroke elimination network used to remove text strokes in an image, the second stroke elimination network may also include a third processing module for increasing the model depth of the second stroke elimination network; and / or, the second stroke elimination network may also include a fourth processing module for increasing the number of model parameters of the second stroke elimination network. Specifically, the second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module; the second stroke elimination network also includes a third processing module and / or a fourth processing module, wherein the third processing module is used to increase the model depth of the second stroke elimination network, and the fourth processing module is used to increase the number of model parameters of the second stroke elimination network; at this time, at least the first target image and the third mask are input into the second stroke elimination network, and the second target image is obtained after processing by the second stroke elimination network, specifically including: at least the first target image and the third mask are input into the second stroke elimination network; the fourth downsampling module downsamples the first target image according to the third mask to obtain the second feature of the first target image; the second feature of the first target image is processed by the third processing module and / or the fourth processing module to obtain the third feature of the first target image; the fourth upsampling module upsamples the third feature of the first target image to obtain the second target image.

[0221] For example, the third processing module can be used to perform convolution processing, residual block processing, dilated convolution processing, pooling processing, inverse residual block processing, activation function processing, etc., and / or, the second stroke elimination network can also include a fourth processing module for increasing the number of model parameters of the second stroke elimination network. For example, the fourth processing module can be used to perform convolution processing, dilated convolution processing, residual block processing, etc., thereby improving the text elimination effect of the first stroke elimination network when eliminating text strokes through the third processing module and / or the fourth processing module, and further improving the rendering backfill effect during the text elimination process.

[0222] Similar to the implementation process of the first weighted discriminator in step S103, where the first weighted discriminator is a discriminator corresponding to a generator composed of algorithm units, this application designs another weighted discriminator for the generator (i.e., two algorithm units) composed of the first stroke detection network, the second stroke elimination network, and the second stroke detection network and the second stroke elimination network. After the second stroke elimination network processes the second target image, the method further includes: inputting the second target image and its ground truth image into the second weighted discriminator; after processing by the second weighted discriminator, a second evaluation result is obtained. This second evaluation result indicates the difference between the second target image and its ground truth image. The subsequent second stroke elimination network can update its training parameters using this second evaluation result, further improving the text elimination effect of the second stroke elimination network.

[0223] Specifically, the second weighted discriminator may include:

[0224] Third identification formula:

[0225]

[0226] Fourth identification formula:

[0227]

[0228] Where, label ijk This refers to the value of the (i,j,k)th element in the patch, where ω2 is the weight of that element, and n×n×C is the size of the feature block (patch).

[0229] In this embodiment, the second weighted discriminator can be implemented using the third and fourth discrimination formulas. The authenticity of the second target image, where the proportion of character strokes is larger, has a greater impact on the final result of the first weighted discriminator. Furthermore, the implementation process of the second weighted discriminator can refer to the aforementioned implementation process of the first weighted discriminator, and will not be repeated here.

[0230] For example, please refer to Figure 5-3 This is the generator network for the character elimination algorithm based on character stroke detection proposed in this application. (Above) Figure 5-2 The content shown has constructed the first stroke detection network G. D And the first stroke elimination network G R Here, a similar second stroke detection network G' is constructed. D And a similar second-stroke elimination network G' RIt is important to note that the inputs to the two networks constructed here differ from those of the previous networks. Specifically, the second stroke detection network G' D The input consists of three parts: the first stroke elimination network G in the first algorithm unit. R Output I te The output of the first stroke detection network in the first algorithm unit is M. s G' D The purpose is to analyze G in the first algorithm unit. D The output of G is further optimized because D The detected stroke masks still exhibit varying degrees of inaccuracy, or missegmentation, at the edges of character strokes. The second stroke elimination network G' in the second algorithm unit... R The input can consist of the following three parts: the character stroke elimination network G in the first algorithm unit. R Output I te The second algorithm unit is the character stroke detection network G'. D The output M' s and mask.

[0231] Please see Figure 7 This application provides another image processing method, including:

[0232] S201, Step 1: Determine the first mask corresponding to the original image based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, and the target text is composed of character strokes.

[0233] In this embodiment, step S201 can refer to the foregoing. Figure 4 The implementation process of step S101 in the illustrated embodiment will not be described in detail here.

[0234] S202, Step 2: Input the original image and the first mask into the first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the text strokes in the original image. The first stroke detection network is used to detect text strokes in the image.

[0235] In one possible implementation, in step 2, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module. Specifically, the process of inputting the original image and the first mask into the first stroke detection network in step 2, and obtaining the second mask after processing by the first stroke detection network, can include: inputting the original image and the first mask into the first stroke detection network; then, the first downsampling module downsamples the original image according to the first mask to obtain a first feature of the original image; subsequently, the first upsampling module upsamples the first feature of the original image to obtain the second mask. The first stroke detection network and... Figure 4 The implementation of the first stroke detection network, which includes a first downsampling module and a first upsampling module, is similar in the embodiment and will not be described again here.

[0236] In another possible implementation, in step 2, the first stroke detection network used to detect character strokes in the image can be a U-shaped network. Besides including a first downsampling module and a first upsampling module, the first stroke detection network can further include a fifth processing module and / or a sixth processing module. The fifth processing module is used to increase the model depth of the first stroke detection network, and the sixth processing module is used to increase the number of model parameters of the first stroke detection network. Therefore, in step S102, inputting the original image and the first mask into the first stroke detection network, and obtaining the second mask after processing by the first stroke detection network, specifically includes: inputting the original image and the first mask into the first stroke detection network; the first downsampling module downsampling the original image according to the first mask to obtain a fourth feature of the original image; the fourth feature of the original image being processed by the fifth processing module and / or the sixth processing module to obtain a fifth feature of the original image; and the first upsampling module upsampling the fifth feature of the original image to obtain the second mask. The first stroke detection network and... Figure 4 The implementation of the first stroke detection network, which includes a first downsampling module, a first upsampling module, a fifth processing module, and / or a sixth processing module, is similar in the embodiment and will not be described again here.

[0237] In one possible implementation, in step 2, the first stroke detection network further includes a first loss function, wherein the first loss function includes:

[0238]

[0239] Where M is the first mask, M s For the second mask, M gt For M s The corresponding truth value, λ d For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network.

[0240] Wherein, the first loss function and Figure 4 The implementation of the first loss function in the embodiments is similar and will not be described again here.

[0241] In this embodiment, step S202 can be referred to the foregoing. Figure 4 The implementation process of step S102 in the illustrated embodiment will not be described in detail here.

[0242] S203. Determine whether the first preset condition is met. If yes, proceed to step S204; otherwise, proceed to step S202.

[0243] In this embodiment, the first preset condition can be a pre-configured value or a value for the number of executions, execution duration, execution cycle, or other parameters determined in response to user input instructions, and is not limited here.

[0244] For example, when the first preset condition is pre-configuration, the smart terminal can directly pre-configure "execution count value, execution duration value, execution cycle value, or other parameters" in its own storage space. The terminal device can also pre-store the mapping relationship between the original image parameters (resolution, pixel depth, pixel size, etc.) and "execution count value, execution duration value, execution cycle value, or other parameters". When the first preset condition is determined in response to user input instructions, the smart terminal can directly use the data input by the user through the input device (mouse, keyboard, touch screen, etc.) to determine "execution count value, execution duration value, execution cycle value, or other parameters".

[0245] Here, we take the execution count as an example. For instance, if the execution count is 3 times, when the smart terminal determines that step S202 (i.e., step 2) has been executed once, it determines that the first preset condition is not met. At this time, the second mask replaces the first mask, and step S202 (i.e., step 2) is executed again. When the smart terminal determines that step S202 (i.e. step 2) has been executed twice, it determines that the first preset condition is not met. At this time, the second mask replaces the first mask, and step S202 (i.e. step 2) is executed again. When the smart terminal determines that step S202 (i.e. step 2) has been executed three times, it determines that the first preset condition is met. At this time, the smart terminal uses the second mask obtained from the result of executing step S202 (i.e. step 2) for the third execution as the input for step S204 (i.e. step 3), and executes step S204 (i.e. step 3).

[0246] S204, Step 3: Input at least the original image and the second mask into the first stroke elimination network. After processing by the first stroke elimination network, a first target image is obtained. The first stroke elimination network is used to eliminate text strokes in the image. The first target image is the image after text elimination of the original image.

[0247] In this embodiment, step S203 can be referred to the foregoing. Figure 4 The implementation process of step S103 in the illustrated embodiment differs in that the second mask in the input data comes from step S203, and the implementation process will not be described in detail here.

[0248] In one possible implementation, in step 3, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module. Specifically, the process of inputting at least the original image and the second mask into the first stroke elimination network in step 3 to obtain the first target image after processing by the first stroke elimination network can include: inputting at least the original image and the second mask into the first stroke elimination network; then, the second downsampling module downsamples the original image according to the second mask to obtain a second feature of the original image; subsequently, the second upsampling module upsamples the second feature of the original image to obtain the first target image. The first stroke elimination network and... Figure 4 The implementation of the first stroke elimination network in the embodiment is similar and will not be described again here.

[0249] In one possible implementation, the process of inputting at least the original image and the second mask into the first stroke elimination network in step 3, and obtaining the first target image after processing by the first stroke elimination network, may specifically include: inputting the original image, the first mask, and the second mask into the first stroke elimination network, and obtaining the first target image after processing by the first stroke elimination network.

[0250] In one possible implementation, in step 3, the second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module; the second stroke elimination network also includes a third processing module and / or a fourth processing module, wherein the third processing module is used to increase the model depth of the second stroke elimination network, and the fourth processing module is used to increase the number of model parameters of the second stroke elimination network; at least the first target image and the third mask are input into the second stroke elimination network, and the second target image is obtained after processing by the second stroke elimination network specifically includes: at least the first target image and the third mask are input into the second stroke elimination network; the fourth downsampling module downsamples the first target image according to the third mask to obtain the second feature of the first target image; the second feature of the first target image is processed by the third processing module and / or the fourth processing module to obtain the third feature of the first target image; the fourth upsampling module upsamples the third feature of the first target image to obtain the second target image. The first processing module and the second processing module are related to... Figure 4 The implementation of the first processing module and the second processing module in the embodiment is similar, and will not be described again here.

[0251] In one possible implementation, in step 3, the first stroke elimination network further includes a second loss function, wherein the second loss function includes:

[0252]

[0253] Among them, I te For the first target image, I gt For I te The corresponding truth value, λ r For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the character stroke elimination network. The second loss function is... Figure 4 The implementation of the second loss function in the embodiments is similar and will not be described again here.

[0254] Furthermore, after step S204, the smart terminal can determine whether to execute step S205 or step S206 based on pre-configuration or in response to user input instructions.

[0255] S205. Determine whether the second preset condition is met. If yes, proceed to step S207; otherwise, proceed to step S204.

[0256] In this embodiment, similar to the implementation of the first preset condition, the second preset condition can be a pre-configured value or a value for the number of executions, execution duration, execution cycle, or other parameters determined in response to user input instructions, which is not limited here.

[0257] Here, we will still take the execution count as the first preset condition as an example. For example, when the execution count is 2 times, when the smart terminal determines that step S204 (i.e. step 3) has been executed once, it determines that the first preset condition is not met. At this time, the first target image will replace the original image, and step S204 (i.e. step 3) will be executed again. When the smart terminal determines that step S204 (i.e. step 3) has been executed twice, it determines that the second preset condition is met. At this time, the smart terminal executes step S207.

[0258] S206. Determine whether the third preset condition is met. If yes, proceed to step S207; otherwise, proceed to step S204.

[0259] In this embodiment, similar to the implementation of the first and second preset conditions, the third preset condition can be a pre-configured value or a value for the number of executions, execution duration, execution cycle, or other parameters determined in response to user input instructions, which is not limited here.

[0260] Here, we will still take the execution count as the first preset condition as an example. For example, when the execution count is 2 times, when the smart terminal determines that step S204 (i.e. step 3) has been executed once, it determines that the first preset condition is not met. At this time, the first target image replaces the original image, and steps S202 (i.e. step 2) and S204 (i.e. step 3) are executed again. When the smart terminal determines that steps S202 (i.e. step 2) and S204 (i.e. step 3) have been executed twice, it determines that the third preset condition is met. At this time, the smart terminal executes step S207.

[0261] S207, Perform other operations.

[0262] In this embodiment, when the smart terminal determines in step S205 that the second preset condition is met, or when the smart terminal determines in step S206 that the third preset condition is met, the smart terminal can execute other steps, specifically including:

[0263] 1) Output the first target image obtained in step S205 or step S206;

[0264] 2) Step 4: Input the first target image and the ground truth image of the first target image into the first weighted discriminator. After processing by the first weighted discriminator, a first evaluation result is obtained. The first evaluation result is used to indicate the difference between the first target image and the ground truth image of the first target image.

[0265] In one possible implementation, in step 4, the first weighted discriminator includes:

[0266] First identification formula:

[0267]

[0268] Second identification formula:

[0269]

[0270] Where, label ijk This refers to the value of the (i,j,k)th element in the patch, where ω1 is the weight of that element, and n×n×C is the size of the feature block (patch).

[0271] Among them, the first weighted discriminator and Figure 4 The implementation of the first weighted discriminator in the embodiment is similar and will not be described again here.

[0272] based on Figures 4 to 7 As illustrated in the embodiments, this application verifies the effectiveness of the constructed network through a series of experiments. Since there is currently no dataset based on real-world data for text removal tasks, this application constructs a test example. In this test example, the constructed dataset consists of 10,000 images with different languages ​​and texts in different scenes. To ensure the reasonableness of the results, this application randomly selects 1,000 images from the dataset as the test set, and the remainder as the training set. The experimental platform is a 64-bit Ubuntu server with 16 Intel Xeon E5-2690v4 2.60GHz CPUs, 256GB RAM, and 8 TITANRTX graphics cards. The performance of the convolutional neural network designed in this application is verified on this platform using the TensorFlow 1.13.1 deep learning framework.

[0273] First, the effectiveness of the image processing method based on character stroke detection proposed in this application is verified. This application uses a network with only one character stroke elimination network as the generator and a typical dense image patch spectral-normalized discriminator (SNPatchGAN) as the discriminator as the baseline network. The effectiveness of these modules is verified by adding or replacing different modules from the proposed image processing method on the baseline network. The effectiveness of this module is verified by adding the character stroke detection network TSDNet to the baseline network; the module's function is verified by replacing the discriminator of the baseline network with the patch-based weighted discriminator (WD) proposed in this application; and the superiority of the proposed complete network Cascade over the aforementioned network variants is also verified. Table 1 shows the quantization results of the four networks on the test set after training and convergence. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) are used to evaluate the text removal results. The mean absolute error between the detected text stroke mask and the ground-truth stroke mask (tMAE) is used to evaluate the stroke mask; lower MAE and tMAE indicate better network performance, while lower PSNR and SSIM indicate the opposite. The image processing method based on text stroke detection proposed in this application outperforms other network variants, demonstrating that introducing a text stroke detection network and a patch-based weighted discriminator can significantly improve the text removal performance. To more intuitively illustrate the text removal effects of different network variants, such as... Figure 8 Examples are shown. It can be seen that Cascade, the complete network proposed in this application, can better preserve the texture, structure and other information in the image while performing text removal, making the image more natural.

[0274] Method MAE (%) PSNR SSIM tMAE (%) Baseline 1.59 35.00 0.9542 - WD 1.00 38.31 0.9722 - TSDNet 0.98 38.17 0.9733 7.63 Cascade 0.75 39.44 0.9756 4.73

[0275] Table 1

[0276] Secondly, experiments were conducted to demonstrate the superiority of the image processing method proposed in this application compared to existing text removal algorithms. This application selected three state-of-the-art deep learning-based text removal algorithms for comparison: STEraser, EnsNet, and MTRNet. Table 2 shows the comparison results of the image processing method in this application with the other three algorithms on relevant datasets. MAE, PSNR, and SSIM are evaluation metrics for the image quality after text removal, while RPF is used to re-detect text in the image after text removal; the fewer characters detected, the better the text removal effect. Ideally, all three metrics should be 0.

[0277]

[0278]

[0279] Table 2

[0280] Comparing the results in Table 2, it can be seen that the image processing method designed in this application generates images with the best quality and achieves the most thorough text removal, with all relevant indicators far exceeding those of other latest text removal algorithms. This demonstrates that the algorithm designed in this application is effective.

[0281] Finally, the network proposed in this application can also achieve multilingual text removal and selective text removal. Most existing algorithms can only handle text types that appear in the training set, and cannot handle text types that the algorithm did not encounter during training. The algorithm designed in this application can handle text in different languages ​​very well, showing significant advantages compared to other algorithms. Furthermore, the algorithm proposed in this application can achieve selective text removal; that is, by modifying the mask, only a portion of the text contained in the image can be removed, while other parts of the text remain unchanged.

[0282] In summary, the image processing method provided in this application can achieve better text removal results by performing precise detection and segmentation of text in the image.

[0283] exist Figures 1 to 8 Based on the corresponding embodiments, in order to better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above-described solutions is also provided below.

[0284] Please refer to details. Figure 9 , Figure 9 This is a schematic diagram of the structure of an image processing apparatus 900 provided in an embodiment of this application.

[0285] In one specific implementation, the image processing apparatus 900 includes:

[0286] The determining unit 901 is used to determine a first mask corresponding to the original image based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, which is composed of character strokes.

[0287] The first processing unit 902 is used to input the original image and the first mask into the first stroke detection network, and obtain the second mask after processing by the first stroke detection network. The second mask is used to indicate the position of the character stroke in the original image. The first stroke detection network is used to detect character strokes in the image.

[0288] The second processing unit 903 is used to input at least the original image and the second mask into the first stroke elimination network, and obtain a first target image after processing by the first stroke elimination network. The first stroke elimination network is used to eliminate text strokes in the image, and the first target image is the image after text elimination of the original image.

[0289] In this embodiment, the first stroke detection network is used to detect text strokes in the image, and the first stroke elimination network is used to eliminate text strokes in the image. That is, the first processing unit 902 detects the text strokes in the original image through the first stroke detection network and obtains a second mask indicating the position of the text strokes in the original image. Then, the second processing unit 903 performs text elimination in the original image based on the second mask through the first stroke elimination network to obtain the first target image. That is, the first target image is obtained by performing text elimination in the original image with the mask indicating the position of the text strokes in the original image as the granularity of text elimination. Compared with the text elimination process implemented with the rectangular mask where the text is located as the granularity, it avoids the loss of image information in the text background of the rectangular mask, thereby increasing the coherence of image information in the image after text elimination to a certain extent and improving the rendering backfilling effect during the text elimination process.

[0290] exist Figure 9 Based on the illustrated embodiment, the image processing apparatus 900 can further implement other embodiments; please refer to [link to relevant documentation]. Figure 10 The image processing apparatus 900 can also be used to perform at least one of the following embodiments, including:

[0291] In one possible implementation, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module;

[0292] The first processing unit 902 is specifically used for:

[0293] The original image and the first mask are input into the first stroke detection network;

[0294] The first downsampling module downsamples the original image based on the first mask to obtain the first feature of the original image;

[0295] The first upsampling module upsamples the first feature of the original image to obtain the second mask.

[0296] In one possible implementation,

[0297] The first stroke elimination network is a U-shaped network, which includes a second downsampling module and a second upsampling module;

[0298] The second processing unit 903 is specifically used for:

[0299] At least the original image and the second mask are input into the first stroke elimination network;

[0300] The second downsampling module downsamples the original image based on the second mask to obtain the second feature of the original image;

[0301] The second upsampling module upsamples the second feature of the original image to obtain the first target image.

[0302] In one possible implementation, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module; the first stroke detection network also includes a fifth processing module and / or a sixth processing module, wherein the fifth processing module is used to increase the model depth of the first stroke detection network, and the sixth processing module is used to increase the number of model parameters of the first stroke detection network;

[0303] The second processing unit 903 is specifically used for:

[0304] The original image and the first mask are input into the first stroke detection network; the first downsampling module downsamples the original image according to the first mask to obtain the fourth feature of the original image; the fourth feature of the original image is processed by the fifth processing module and / or the sixth processing module to obtain the fifth feature of the original image; the first upsampling module upsamples the fifth feature of the original image to obtain the second mask.

[0305] In one possible implementation, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; the first stroke elimination network also includes a first processing module and / or a second processing module, wherein the first processing module is used to increase the model depth of the first stroke elimination network, and the second processing module is used to increase the number of model parameters of the first stroke elimination network;

[0306] The second processing unit 903 is specifically used for:

[0307] The original image and the second mask are input into the first stroke elimination network at least once;

[0308] The second downsampling module downsamples the original image based on the second mask to obtain the second feature of the original image;

[0309] The second feature of the original image is processed by the first processing module and / or the second processing module to obtain the third feature of the original image;

[0310] The second upsampling module upsamples the third feature of the original image to obtain the first target image.

[0311] In one possible implementation, the second processing unit 903 is specifically used for:

[0312] The original image, the first mask, and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network.

[0313] In one possible implementation, the device 900 further includes:

[0314] The third processing unit 904 is used to input at least the first target image and the second mask into the second stroke detection network, and obtain the third mask after processing by the second stroke detection network. The second stroke detection network is used to detect text strokes in the image.

[0315] The fourth processing unit 905 is used to input at least the first target image and the third mask into the second stroke elimination network, and obtain the second target image after processing by the second stroke elimination network. The second stroke elimination network is used to eliminate text strokes in the image.

[0316] In one possible implementation,

[0317] The second stroke detection network is a U-shaped network, which includes a third downsampling module and a third upsampling module;

[0318] The third processing unit 904 is specifically used for:

[0319] At least the first target image and the second mask are input into the second stroke detection network;

[0320] The third downsampling module downsamples the first target image based on the second mask to obtain the first feature of the first target image;

[0321] The third upsampling module upsamples the first feature of the first target image to obtain the third mask.

[0322] In one possible implementation, the second stroke detection network is a U-shaped network, including a third downsampling module and a third upsampling module; the second stroke detection network also includes a seventh processing module and / or an eighth processing module, wherein the seventh processing module is used to increase the model depth of the second stroke detection network, and the eighth processing module is used to increase the number of model parameters of the second stroke detection network.

[0323] The third processing unit 904 is specifically used for:

[0324] The first target image and the second mask are input into the second stroke detection network; the third downsampling module downsamples the original image according to the second mask to obtain the sixth feature of the original image; the sixth feature of the original image is processed by the seventh processing module and / or the eighth processing module to obtain the seventh feature of the original image; the third upsampling module upsamples the seventh feature of the original image to obtain the third mask.

[0325] In one possible implementation,

[0326] The second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module;

[0327] The fourth processing unit 905 is specifically used for:

[0328] At least the first target image and the third mask are input into the second stroke elimination network;

[0329] The fourth downsampling module downsamples the first target image based on the third mask to obtain the second feature of the first target image;

[0330] The fourth upsampling module upsamples the second feature of the first target image to obtain the second target image.

[0331] In one possible implementation, the second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module; the second stroke elimination network also includes a third processing module and / or a fourth processing module, wherein the third processing module is used to increase the model depth of the second stroke elimination network, and the fourth processing module is used to increase the number of model parameters of the second stroke elimination network.

[0332] The fourth processing unit 905 is specifically used for:

[0333] At least the first target image and the third mask are input into the second stroke elimination network;

[0334] The fourth downsampling module downsamples the first target image based on the third mask to obtain the second feature of the first target image;

[0335] The second feature of the first target image is processed by the third processing module and / or the fourth processing module to obtain the third feature of the first target image;

[0336] The fourth upsampling module upsamples the third feature of the first target image to obtain the second target image.

[0337] In one possible implementation, the third processing unit 904 is specifically used for:

[0338] The first target image, the first mask, and the second mask are input into the second stroke detection network, and the third mask is obtained after processing by the second stroke detection network.

[0339] In one possible implementation, the fourth processing unit 905 is specifically used for:

[0340] The first target image, the first mask, and the third mask are input into the second stroke elimination network, and the second target image is obtained after processing by the second stroke elimination network.

[0341] In one possible implementation, the first stroke detection network further includes a first loss function, wherein the first loss function comprises:

[0342]

[0343] Where M is the first mask, M s For the second mask, M gt For M s The corresponding truth value, λ d For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network.

[0344] In one possible implementation, the first stroke elimination network further includes a second loss function, wherein the second loss function comprises:

[0345]

[0346] Among them, I te For the first target image, I gt For I te The corresponding truth value, λ r For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the character stroke elimination network.

[0347] In one possible implementation, the device 900 further includes:

[0348] The fifth processing unit 906 is used to input the first target image and the ground truth image of the first target image into the first weighted discriminator, and obtain a first evaluation result after processing by the first weighted discriminator. The first evaluation result is used to indicate the difference between the first target image and the ground truth image of the first target image.

[0349] In one possible implementation, the device further includes:

[0350] The sixth processing unit 907 is used to input the second target image and the ground truth image of the second target image into the second weighted discriminator, and obtain a second evaluation result after processing by the second weighted discriminator. The second evaluation result is used to indicate the difference between the second target image and the ground truth image of the second target image.

[0351] The components of the image processing device 900 can also be used to execute the steps performed in the various possible implementations of the aforementioned method embodiments. For details, please refer to the aforementioned method embodiments, which will not be repeated here.

[0352] In another specific implementation, the image processing apparatus 900 includes:

[0353] The determining unit 901 is used to determine the first mask corresponding to the original image in step 1 based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, which is composed of character strokes.

[0354] The first processing unit 902 is used to input the original image and the first mask into the first stroke detection network in step 2, and obtain the second mask after processing by the first stroke detection network. The second mask is used to indicate the position of the character stroke in the original image, and the first stroke detection network is used to detect the character stroke in the image.

[0355] The second processing unit 903 is used to input the original image and the second mask into the first stroke elimination network in step 3, and obtain the first target image after processing by the first stroke elimination network. The first stroke elimination network is used to eliminate text strokes in the image, and the first target image is the image after text elimination of the original image.

[0356] The first processing unit 902 is further configured to at least replace the first mask with the second mask and repeat step 2 until the first preset condition is met; and / or, the second processing unit 903 is further configured to replace the original image with the first target image and repeat step 3 until the second preset condition is met; and / or, the first processing unit 902 and the second processing unit 903 are further configured to replace the original image with the first target image and repeat steps 2 and 3 until the third preset condition is met.

[0357] In this embodiment, the first stroke detection network in step 2 of the first processing unit 902 is used to detect text strokes in the image, and the first stroke elimination network in step 3 of the second processing unit 903 is used to eliminate text strokes in the image. That is, after the first processing unit 902 detects the text strokes in the original image through the first stroke detection network in step 2 and obtains a second mask indicating the position of the text strokes in the original image, the second processing unit 903 performs text elimination in the original image according to the second mask in step 3 to obtain a first target image. That is, the first target image is obtained by performing text elimination in the original image with the mask indicating the position of the text strokes in the original image as the granularity of text elimination. Compared with the text elimination process implemented with the rectangular mask where the text is located as the granularity, the loss of image information in the text background of the rectangular mask is avoided. That is, by doing so, the continuity of image information in the image after text elimination is increased to a certain extent, and the rendering backfilling effect in the text elimination process is improved. In addition, repeating steps 2 and / or 3 until the preset conditions (first preset condition, second preset condition and / or third preset condition) are met can improve the stroke detection effect and stroke elimination effect to a certain extent through the repeated execution of the first stroke detection network and the second stroke elimination network, and further improve the rendering backfill effect during the text elimination process.

[0358] It should be noted that the preset conditions (first preset condition, second preset condition and / or third preset condition) can be pre-configured or determined in response to user input commands, such as the number of executions, execution duration, execution cycle, or other parameters, which are not limited here.

[0359] exist Figure 9 Based on the illustrated embodiment, the image processing apparatus 900 can further implement other embodiments; please refer to [link to relevant documentation]. Figure 10 The image processing apparatus 900 can also be used to perform at least one of the following embodiments, including:

[0360] In one possible implementation, the first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module;

[0361] The first processing unit 902 is specifically used in step 2 to:

[0362] The original image and the first mask are input into the first stroke detection network;

[0363] The first downsampling module downsamples the original image based on the first mask to obtain the first feature of the original image;

[0364] The first upsampling module upsamples the first feature of the original image to obtain the second mask.

[0365] In one possible implementation,

[0366] The first stroke elimination network is a U-shaped network, which includes a second downsampling module and a second upsampling module;

[0367] The second processing unit 903 is specifically used in step 3 to:

[0368] The original image and the second mask are input into the first stroke elimination network;

[0369] The second downsampling module downsamples the original image based on the second mask to obtain the second feature of the original image;

[0370] The second upsampling module upsamples the second feature of the original image to obtain the first target image.

[0371] In one possible implementation, the first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; the first stroke elimination network also includes a first processing module and / or a second processing module, wherein the first processing module is used to increase the model depth of the first stroke elimination network, and the second processing module is used to increase the number of model parameters of the first stroke elimination network;

[0372] The second processing unit 903 is specifically used for:

[0373] The original image and the second mask are input into the first stroke elimination network at least once;

[0374] The second downsampling module downsamples the original image based on the second mask to obtain the second feature of the original image;

[0375] The second feature of the original image is processed by the first processing module and / or the second processing module to obtain the third feature of the original image;

[0376] The second upsampling module upsamples the third feature of the original image to obtain the first target image.

[0377] In one possible implementation, the second processing unit 903 is specifically used in step 3 to:

[0378] The original image, the first mask, and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network.

[0379] In one possible implementation, the first stroke detection network further includes a first loss function, wherein the first loss function comprises:

[0380]

[0381] Where M is the first mask, M s For the second mask, M gt For M s The corresponding truth value, λ d For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network.

[0382] In one possible implementation, the first stroke elimination network further includes a second loss function, wherein the second loss function comprises:

[0383]

[0384] Among them, I te For the first target image, I gt For I te The corresponding truth value, λ r For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the character stroke elimination network.

[0385] In one possible implementation, the device 900 further includes:

[0386] The fifth processing unit 906 is used to input the first target image and the ground truth image of the first target image into the first weighted discriminator in step 4, and obtain a first evaluation result after processing by the first weighted discriminator. The first evaluation result is used to indicate the difference between the first target image and the ground truth image of the first target image.

[0387] The components of the image processing device 900 can also be used to execute the steps performed in the various possible implementations of the aforementioned method embodiments. For details, please refer to the aforementioned method embodiments, which will not be repeated here.

[0388] This application also provides an image processing apparatus; please refer to [link to relevant documentation]. Figure 11 , Figure 11This is a schematic diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 1100 may be equipped with... Figure 9 The image processing apparatus 900 described in the corresponding embodiment is used to implement the functions of the image processing apparatus in the foregoing method embodiment; alternatively, the image processing apparatus 1100 may be equipped with... Figure 9 The image processing apparatus 900 described in the corresponding embodiment is used to implement the functions of the image processing apparatus in the foregoing method embodiment. Specifically, the image processing apparatus 1100 is implemented by one or more image processing devices. The image processing apparatus 1100 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1123 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 can be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the image processing apparatus. Furthermore, the central processing unit 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the image processing apparatus 1100. However, it should be understood that... Figure 11 The image processing device shown is only one example of an image processing device. The image processing device 1100 may not include the memory 1132 and the storage medium 1130. Instead, an external memory may be configured outside the image processing device 1100. That is, the memory 1132 and the storage medium 1130 and the central processing unit 1122 can be independent devices. For example, an external memory is used in a vehicle image processing device.

[0389] The image processing device 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0390] In this embodiment, the central processing unit 1122 is used to execute the image processing method executed by the image processing device in the foregoing method embodiments. It should be noted that the specific implementation of the image processing method executed by the central processing unit 1122 can be referred to the descriptions in the foregoing method embodiments, and will not be repeated here.

[0391] This application also provides a computer program product that, when run on a computer, causes the computer to perform the steps executed by the image processing apparatus in the method described in the foregoing method embodiments.

[0392] This application also provides a computer-readable storage medium storing a program for performing signal processing, which, when run on a computer, causes the computer to execute the steps performed by the image processing apparatus in the method described in the foregoing method embodiments.

[0393] This application provides a circuit system including a processing circuit configured to perform the image processing method described in the foregoing method embodiments.

[0394] The execution device, training device, smart terminal, or communication device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the image processing device to execute the image processing method described in the above method embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0395] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0396] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0397] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0398] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. An image processing method, characterized in that, include: Based on the original image, a first mask corresponding to the original image is determined. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, which is composed of character strokes. The original image and the first mask are input into the first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the text strokes in the original image. The position information indicated by the second mask is the position identified by the pixel coordinate system. The first stroke detection network is used to detect text strokes in the image. The original image and the second mask are input into the first stroke removal network. After processing by the first stroke removal network, a first target image is obtained. The first stroke removal network is used to remove text strokes in the image. The first target image is the image after text removal of the original image. After at least the original image and the second mask are input into the first stroke removal network, and the first target image is obtained after processing by the first stroke removal network, the method further includes: The first target image and the second mask are input into the second stroke detection network. After processing by the second stroke detection network, a third mask is obtained. The second stroke detection network is used to detect text strokes in the image. The first target image and the third mask are input into the second stroke elimination network. After processing by the second stroke elimination network, a second target image is obtained. The second stroke elimination network is used to eliminate text strokes in the image.

2. The method according to claim 1, characterized in that, The first stroke detection network is a U-shaped network, including a first downsampling module and a first upsampling module; The original image and the first mask are input into the first stroke detection network. After processing by the first stroke detection network, the second mask is obtained, specifically including: The original image and the first mask are input into the first stroke detection network; The first downsampling module downsamples the original image based on the first mask to obtain the first feature of the original image; The first upsampling module upsamples the first feature of the original image to obtain the second mask.

3. The method according to claim 1 or 2, characterized in that, The first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; The step of inputting at least the original image and the second mask into the first stroke elimination network, and obtaining the first target image after processing by the first stroke elimination network, specifically includes: The original image and the second mask are at least input into the first stroke elimination network; The second downsampling module downsamples the original image based on the second mask to obtain the second feature of the original image; The second upsampling module upsamples the second feature of the original image to obtain the first target image.

4. The method according to claim 1 or 2, characterized in that, The first stroke elimination network is a U-shaped network, including a second downsampling module and a second upsampling module; the first stroke elimination network also includes a first processing module and / or a second processing module, wherein the first processing module is used to increase the model depth of the first stroke elimination network, and the second processing module is used to increase the number of model parameters of the first stroke elimination network; The step of inputting at least the original image and the second mask into the first stroke elimination network, and obtaining the first target image after processing by the first stroke elimination network, specifically includes: The original image and the second mask are at least input into the first stroke elimination network; The second downsampling module downsamples the original image based on the second mask to obtain the second feature of the original image; The second feature of the original image is processed by the first processing module and / or the second processing module to obtain the third feature of the original image; The second upsampling module upsamples the third feature of the original image to obtain the first target image.

5. The method according to any one of claims 1 to 2, characterized in that, The step of inputting at least the original image and the second mask into the first stroke removal network, and obtaining the first target image after processing by the first stroke removal network, includes: The original image, the first mask, and the second mask are input into the first stroke elimination network, and the first target image is obtained after processing by the first stroke elimination network.

6. The method according to claim 1, characterized in that, The second stroke detection network is a U-shaped network, which includes a third downsampling module and a third upsampling module; The step of inputting at least the first target image and the second mask into the second stroke detection network, and obtaining the third mask after processing by the second stroke detection network, includes: At least the first target image and the second mask are input into the second stroke detection network; The third downsampling module downsamples the first target image according to the second mask to obtain the first feature of the first target image; The third upsampling module upsamples the first feature of the first target image to obtain the third mask.

7. The method according to claim 1, characterized in that, The second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module; The step of inputting at least the first target image and the third mask into the second stroke removal network, and obtaining the second target image after processing by the second stroke removal network, specifically includes: At least the first target image and the third mask are input into the second stroke elimination network; The fourth downsampling module downsamples the first target image according to the third mask to obtain the second feature of the first target image; The fourth upsampling module upsamples the second feature of the first target image to obtain the second target image.

8. The method according to claim 1, characterized in that, The second stroke elimination network is a U-shaped network, including a fourth downsampling module and a fourth upsampling module; the second stroke elimination network also includes a third processing module and / or a fourth processing module, wherein the third processing module is used to increase the model depth of the second stroke elimination network, and the fourth processing module is used to increase the number of model parameters of the second stroke elimination network; The step of inputting at least the first target image and the third mask into the second stroke removal network, and obtaining the second target image after processing by the second stroke removal network, specifically includes: At least the first target image and the third mask are input into the second stroke elimination network; The fourth downsampling module downsamples the first target image according to the third mask to obtain the second feature of the first target image; The second feature of the first target image is processed by the third processing module and / or the fourth processing module to obtain the third feature of the first target image; The fourth upsampling module upsamples the third feature of the first target image to obtain the second target image.

9. The method according to claim 1, characterized in that, The step of inputting at least the first target image and the second mask into the second stroke detection network, and obtaining the third mask after processing by the second stroke detection network, includes: The first target image, the first mask, and the second mask are input into the second stroke detection network, and the third mask is obtained after processing by the second stroke detection network.

10. The method according to claim 1, characterized in that, The step of inputting at least the first target image and the third mask into the second stroke removal network, and obtaining the second target image after processing by the second stroke removal network, includes: The first target image, the first mask, and the third mask are input into the second stroke elimination network, and the second target image is obtained after processing by the second stroke elimination network.

11. The method according to any one of claims 1 to 2, characterized in that, The first stroke detection network further includes a first loss function, wherein the first loss function includes: Where M is the first mask, M s For the second mask, M gt For M s The corresponding truth value, λ d For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first stroke detection network.

12. The method according to claim 11, characterized in that, The first stroke elimination network further includes a second loss function, wherein the second loss function includes: Among them, I te For the first target image, I gt For I te The corresponding truth value, λ r For integers greater than 1, ⊙ represents element-wise multiplication of the feature maps. This represents the weighted L1 loss function of the first character stroke elimination network.

13. The method according to any one of claims 1 to 2, characterized in that, The method further includes: The first target image and its ground truth image are input into a first weighted discriminator. After processing by the first weighted discriminator, a first evaluation result is obtained. The first evaluation result is used to indicate the difference between the first target image and its ground truth image.

14. The method according to claim 1, characterized in that, The method further includes: The second target image and its ground truth image are input into the second weighted discriminator. After processing by the second weighted discriminator, a second evaluation result is obtained. The second evaluation result is used to indicate the difference between the second target image and its ground truth image.

15. An image processing method, characterized in that, include: Step 1: Based on the original image, determine the first mask corresponding to the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, and the target text is composed of character strokes. Step 2: Input the original image and the first mask into the first stroke detection network. After processing by the first stroke detection network, a second mask is obtained. The second mask is used to indicate the position of the text strokes in the original image. The position information indicated by the second mask is the position identified by the pixel coordinate system. The first stroke detection network is used to detect text strokes in the image. Step 3: Input the original image and the second mask into the first stroke removal network. After processing by the first stroke removal network, a first target image is obtained. The first stroke removal network is used to remove text strokes in the image. The first target image is the image after text removal of the original image. After at least the original image and the second mask are input into the first stroke removal network, and the first target image is obtained after processing by the first stroke removal network, the method further includes: The first target image and the second mask are input into the second stroke detection network. After processing by the second stroke detection network, a third mask is obtained. The second stroke detection network is used to detect text strokes in the image. The first target image and the third mask are input into the second stroke elimination network. After processing by the second stroke elimination network, the second target image is obtained. The second stroke elimination network is used to eliminate text strokes in the image. Replace the first mask with the second mask and repeat step 2 until the first preset condition is met; and / or replace the original image with the first target image and repeat step 3 until the second preset condition is met; and / or replace the original image with the first target image and repeat steps 2 and 3 until the third preset condition is met.

16. An image processing apparatus, characterized in that, include: The determining unit is configured to determine a first mask corresponding to the original image based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, and the target text is composed of character strokes. The first processing unit is used to input the original image and the first mask into the first stroke detection network, and obtain the second mask after processing by the first stroke detection network. The second mask is used to indicate the position of the text stroke in the original image. The position information indicated by the second mask is the position identified by the pixel coordinate system. The first stroke detection network is used to detect text strokes in the image. The second processing unit is used to input at least the original image and the second mask into the first stroke elimination network, and obtain a first target image after processing by the first stroke elimination network. The first stroke elimination network is used to eliminate text strokes in the image, and the first target image is the image after text elimination of the original image. The third processing unit is used to input at least the first target image and the second mask into the second stroke detection network, and obtain the third mask after processing by the second stroke detection network. The second stroke detection network is used to detect text strokes in the image. The fourth processing unit is used to input at least the first target image and the third mask into the second stroke elimination network, and obtain the second target image after processing by the second stroke elimination network. The second stroke elimination network is used to eliminate text strokes in the image.

17. An image processing apparatus, characterized in that, include: The determining unit is used in step 1 to determine a first mask corresponding to the original image based on the original image. The first mask is used to indicate the position of the target text in the original image. The original image includes the target text, and the target text is composed of character strokes. The first processing unit is used to input the original image and the first mask into the first stroke detection network in step 2, and obtain the second mask after processing by the first stroke detection network. The second mask is used to indicate the position of the text stroke in the original image. The position information indicated by the second mask is the position identified by the pixel coordinate system. The first stroke detection network is used to detect text strokes in the image. The second processing unit is used to input the original image and the second mask into the first stroke elimination network in step 3, and obtain the first target image after processing by the first stroke elimination network. The first stroke elimination network is used to eliminate text strokes in the image, and the first target image is the image after text elimination of the original image. The third processing unit is used to input at least the first target image and the second mask into the second stroke detection network, and obtain the third mask after processing by the second stroke detection network. The second stroke detection network is used to detect text strokes in the image. The fourth processing unit is used to input at least the first target image and the third mask into the second stroke elimination network, and obtain the second target image after processing by the second stroke elimination network. The second stroke elimination network is used to eliminate text strokes in the image. The first processing unit is further configured to replace the first mask with the second mask and repeat step 2 until a first preset condition is met; and / or, the second processing unit is further configured to replace the original image with the first target image and repeat step 3 until a second preset condition is met; and / or, the first processing unit and the second processing unit are further configured to replace the original image with the first target image and repeat steps 2 and 3 until a third preset condition is met.

18. An image processing apparatus, characterized in that, The device includes a processor coupled to a memory storing program instructions that, when executed by the processor, cause the device to perform the method of any one of claims 1 to 14, or, when executed by the processor, perform the method of claim 15.

19. A computer-readable storage medium, characterized in that, The computer storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 14, or the program instructions that, when executed by a processor, cause the processor to perform the method as described in claim 15.

20. A chip system, characterized in that, The chip system includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 14, or to execute the method as described in claim 15.

Images