Image processing method, image processing device, and non-transitory storage medium

Abstract

An image processing method, an image processing device and a non-transitory storage medium. The image processing method comprises: obtaining an input image, wherein the input image comprises printed text and handwritten text; based on the input image, obtaining N groups of initial feature images arranged from high to low in resolution, wherein N is an integer and N>=2; based on the N groups of initial feature images, obtaining an output feature image; based on the output feature image, performing a two-class text detection processing to obtain a printed text detection mask image and a handwritten text detection mask image; and based on the printed text detection mask image and the handwritten text detection mask image, obtaining a first output image, wherein the first output image is an image in which at least part of the handwritten text is erased compared with the input image. Based on the feature pyramid of the N groups of initial feature images, combined with convolution, up-sampling, channel self-attention mechanism convolution and weighted channel fusion operations, the output feature image is obtained.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an image processing method, an image processing apparatus, and a non-transient storage medium. Background Technology

[0002] In educational settings, common teaching materials such as books, test papers, and workbooks often need to be used repeatedly. Common teaching tasks such as book transcription, test paper transcription, and collection of frequently missed questions all require erasing handwritten handwriting and restoring the images for data archiving and repeated practice. Summary of the Invention

[0003] This disclosure provides at least some embodiments of an image processing method, comprising: acquiring an input image, wherein the input image includes printed text and handwritten text; obtaining N initial feature images arranged in descending order of resolution based on the input image, wherein N is a positive integer and N≥2; obtaining an output feature image based on the N initial feature images; performing dual-class text detection processing based on the output feature image to obtain a printed text detection mask image and a handwritten text detection mask image; and obtaining a first output image based on the printed text detection mask image and the handwritten text detection mask image, wherein, compared with the input image, the first output image is an image in which at least a portion of the handwritten text has been erased; wherein obtaining the output feature image based on the N initial feature images includes: upsampling the (i+1)th initial feature image to obtain an image with a resolution equal to or greater than that of the i-th initial feature image. The initial feature images have an i-th group of upsampled feature images with the same resolution. The i-th initial feature image and the i-th upsampled feature image are then concatenated and convolved to obtain the i-th intermediate feature image, where i = 1, 2, ..., N-1. The N-th initial feature image is then convolved to obtain the N-th intermediate feature image. The N-th initial feature image is then convolved using a channel self-attention mechanism to obtain the N+1-th intermediate feature image. The 1-th intermediate feature image is then convolved to obtain the 1-th transition feature image. The j-th intermediate feature image is then convolved and upsampled to obtain the j-th transition feature image, where j = 2, 3, ..., N+1. The N+1-th transition feature images have the same resolution and number of channels. Finally, the N+1-th transition feature images are weighted and fused to obtain the output feature image.

[0004] For example, in some embodiments of the image processing method provided in this disclosure, the weighted channel fusion processing is performed on the N+1 groups of transition feature images to obtain the output feature image, including: performing a pixel-by-pixel and channel-by-channel weighted summation operation on the N+1 groups of transition feature images based on an attention layer to obtain the output feature image, wherein the attention layer includes N+1 nodes, and the N+1 nodes are respectively used to indicate the weights when the N+1 groups of transition feature images are subjected to the weighted summation operation.

[0005] For example, in some embodiments of the image processing methods provided in this disclosure, N≤4.

[0006] For example, in some embodiments of the image processing method provided in this disclosure, N = 3.

[0007] For example, in some embodiments of the image processing method provided in this disclosure, the resolution of the i-th initial feature image is twice the resolution of the (i+1)-th initial feature image, where i = 1, 2, ..., N-1.

[0008] For example, in some embodiments of the image processing method provided in this disclosure, obtaining the N initial feature images arranged from high to low resolution based on the input image includes: performing M consecutive analysis processes on the input image to obtain the M initial feature images arranged from high to low resolution, where M is a positive integer and M≥N; and selecting the N initial feature images from the M initial feature images.

[0009] For example, in some embodiments of the image processing method provided in this disclosure, the printed text detection mask image, the handwritten text detection mask image, the first output image, and the input image have the same resolution.

[0010] For example, in some embodiments of the image processing method provided in this disclosure, obtaining the first output image based on the printed text detection mask image and the handwritten text detection mask image includes: performing a first binarization process on the printed text detection mask image and the handwritten text detection mask image respectively to obtain a printed text binarized image and a handwritten text binarized image; calculating the difference between the printed text binarized image and the handwritten text binarized image to obtain a first mask image; performing a second binarization process on the first mask image to obtain a mask image of the area to be erased; and performing a pixel-by-pixel multiplication operation on the input image and the mask image of the area to be erased to obtain the first output image.

[0011] For example, in some embodiments of the image processing method provided in this disclosure, the first binarization process includes: analyzing the pixel value distribution of the input of the first binarization process to determine a first threshold; and determining the pixel values ​​of the output of the first binarization process according to the relationship between each pixel value of the input of the first binarization process and the first threshold; wherein, when the input of the first binarization process is the printed text detection mask image, the output of the first binarization process is the printed text binarized image; when the input of the first binarization process is the handwritten text detection mask image, the output of the first binarization process is the handwritten text binarized image.

[0012] For example, in some embodiments of the image processing method provided in this disclosure, the second binarization process includes: setting the corresponding pixel value of the mask image of the region to be erased to 1 in response to any pixel value of the first mask image being not less than 0; and setting the corresponding pixel value of the mask image of the region to be erased to 0 in response to any pixel value of the first mask image being less than 0.

[0013] For example, some embodiments of the present disclosure provide an image processing method that further includes: performing simulated background replacement processing on the first output image to obtain a second output image.

[0014] For example, in some embodiments of the image processing method provided in this disclosure, the simulated background replacement processing is performed on the first output image to obtain the second output image, including: blurring the input image to obtain a simulated background image; performing inverse binarization processing on the mask image of the area to be erased to obtain a second mask image; performing a pixel-by-pixel multiplication operation on the simulated background image and the second mask image to obtain a background template image; and performing a pixel-by-pixel addition operation on the first output image and the background template image to obtain the second output image.

[0015] For example, in some embodiments of the image processing method provided in this disclosure, the input image is subjected to the blurring process to obtain the simulated background image, including: performing the blurring process on the input image using a Gaussian blur algorithm to obtain the simulated background image, wherein the blur radius of the Gaussian blur algorithm is set to min(w,h,R), where w and h represent the width and height of the input image, respectively, R is a preset value, w, h, and R are all positive integers, and R≥100.

[0016] For example, some embodiments of the present disclosure provide an image processing method that further includes: performing adaptive light and shadow removal processing on the second output image to obtain a third output image.

[0017] For example, in some embodiments of the image processing method provided in this disclosure, performing the adaptive shading removal processing on the second output image to obtain the third output image includes: performing adaptive threshold binarization processing on the second output image to obtain a third mask image; performing smoothing processing on the third mask image to obtain an intermediate output image; and performing image dilation processing on the intermediate output image to obtain the third output image.

[0018] For example, in some embodiments of the image processing method provided in this disclosure, performing the adaptive threshold binarization processing on the second output image to obtain the third mask image includes: setting the corresponding pixel value of the third mask image to 1 in response to any pixel value in the second output image being not less than the weighted sum of pixel values ​​in its neighboring block; and setting the corresponding pixel value of the third mask image to 0 in response to any pixel value in the second output image being less than the weighted sum of pixel values ​​in its neighboring block.

[0019] For example, in some embodiments of the image processing method provided in this disclosure, the smoothing process is performed on the third mask image to obtain the intermediate output image, including: performing the smoothing process using a closing operation.

[0020] At least some embodiments of this disclosure also provide an image processing apparatus, including: a memory for non-transitory storage of computer-readable instructions; and a processor for executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, perform an image processing method provided in any embodiment of this disclosure.

[0021] At least some embodiments of this disclosure also provide a non-transitory storage medium for storing computer-readable instructions in a non-transitory manner, wherein, when the computer-readable instructions are executed by a computer, they are capable of executing the image processing method provided in any embodiment of this disclosure. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0023] Figure 1 A flowchart illustrating an image processing method provided for at least some embodiments of this disclosure;

[0024] Figure 2 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of steps S200 and S300 shown;

[0025] Figure 3 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of step S200 shown;

[0026] Figure 4 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of step S300 shown;

[0027] Figure 5 A schematic diagram illustrating an input image and its corresponding printed text detection mask image and handwritten text detection mask image provided for at least some embodiments of this disclosure;

[0028] Figure 6 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of step S500 shown;

[0029] Figure 7 A schematic diagram of an input image and its corresponding printed text detection mask image, handwritten text detection mask image and first output image provided for at least some embodiments of this disclosure;

[0030] Figure 8 A flowchart illustrating another image processing method provided for at least some embodiments of this disclosure;

[0031] Figure 9 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 8 An exemplary flowchart of step S600 shown;

[0032] Figure 10 A schematic diagram of an input image and its corresponding first output image, simulated background image and second output image provided for at least some embodiments of this disclosure;

[0033] Figure 11 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 8 An exemplary flowchart of step S700 shown;

[0034] Figure 12 A schematic diagram of a second output image and a third output image provided for at least some embodiments of this disclosure;

[0035] Figure 13 A schematic diagram of an input image and a third output image provided for at least some embodiments of this disclosure;

[0036] Figure 14 A schematic block diagram of an image processing apparatus provided for at least some embodiments of this disclosure; and

[0037] Figure 15 This is a schematic diagram of a non-transient storage medium provided for at least some embodiments of the present disclosure. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0039] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0040] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components are omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is indicated by the same or similar reference numerals in each drawing.

[0041] In educational settings, the following scenarios often occur: (1) Students often need to practice and answer a set of test papers multiple times. In this case, the test paper retains handwritten marks from the previous answers, affecting the effectiveness of subsequent practice. (2) When a student forgets to bring books, test papers, or workbooks, other students can provide photographic images of the books, test papers, or workbooks for the student to print. However, other students' books, test papers, or workbooks may have handwritten notes or answer marks, making it inconvenient for the student to submit subsequent assignments or other learning tasks. (3) Teachers and students often need to digitize important test papers and other learning materials. Taking photos of paper materials is a relatively simple method. However, the drawback of this method is that it is difficult to eliminate draft marks, and the photos are affected by the lighting conditions, resulting in shadows that affect reading and printing. Therefore, it is necessary to design corresponding handwritten mark erasure and image (e.g., test paper images) restoration algorithms for educational scenarios.

[0042] The three main methods currently in use are as follows:

[0043] The first method is based on Optical Character Recognition (OCR) and image pattern matching to locate handwriting. This method uses printed text detection and recognition technology to locate the printed text portion, and then uses image pattern matching to design text patterns, detect all text regions, remove the printed text portion, and treat the remaining portion as handwriting. This pattern matching-based algorithm is relatively complex, making it difficult to design accurate and effective text shape patterns. Furthermore, the size and shape of text vary greatly in educational settings; therefore, this method has poor generalization ability.

[0044] The second method is a handwritten handwriting erasure method based on scene text erasure algorithms. Current research on scene text erasure is quite in-depth; however, text in natural scenes is a low-frequency signal with significant differences between the foreground and background. Furthermore, scene text erasure lacks the ability to distinguish between printed and handwritten text, which can easily lead to large-area accidental erasure of printed text when applied in educational settings.

[0045] The third method involves using handwriting detection combined with color filling of specific image areas to reconstruct the exam paper. This method uses handwriting detection algorithms to accurately locate handwritten parts. However, printed text and handwriting are similar in shape, making it easy for printed text to be mistakenly detected as handwriting, leading to accidental erasure. Furthermore, photographic images are greatly affected by lighting conditions, making it difficult to adapt the fill color of handwritten text to different paper backgrounds, thus affecting the final image's aesthetics.

[0046] In summary, the methods currently used are not effective enough in educational settings, mainly because: (1) handwritten characters are similar in shape to printed characters, making it easy to accidentally erase printed characters. (2) After erasing handwritten characters, the aesthetics of the exam paper image are insufficient. (3) It requires high lighting conditions for the photography environment, and problems such as shadows are easily generated.

[0047] This disclosure provides at least some embodiments of an image processing method. The image processing method includes: acquiring an input image, wherein the input image includes printed text and handwritten text; obtaining N initial feature images arranged in descending order of resolution based on the input image, wherein N is an integer and N≥2; obtaining an output feature image based on the N initial feature images; performing dual-class text detection processing based on the output feature image to obtain a printed text detection mask image and a handwritten text detection mask image; and obtaining a first output image based on the printed text detection mask image and the handwritten text detection mask image, wherein, compared to the input image, the first output image is an image in which at least a portion of the handwritten text has been erased. Based on N initial feature images, the output feature image is obtained, including: upsampling the (i+1)th initial feature image to obtain the i-th upsampled feature image with the same resolution as the i-th initial feature image; concatenating and convolving the i-th initial feature image and the i-th upsampled feature image to obtain the i-th intermediate feature image, where i = 1, 2, ..., N-1; convolving the N-th initial feature image to obtain the N-th intermediate feature image; convolving the N-th initial feature image using a channel self-attention mechanism to obtain the N+1th intermediate feature image; convolving the 1st intermediate feature image to obtain the 1st transition feature image; convolving and upsampling the j-th intermediate feature image to obtain the j-th transition feature image, where j = 2, 3, ..., N+1, and the N+1 transition feature images have the same resolution and number of channels; and weighted channel fusion of the N+1 transition feature images to obtain the output feature image.

[0048] Some embodiments of this disclosure also provide an image processing apparatus corresponding to the above-described image processing method, as well as a non-transient storage medium.

[0049] The image processing method provided in the embodiments of this disclosure can obtain an output feature image based on a feature pyramid formed by N initial feature images, combined with operations such as convolution, upsampling, channel self-attention mechanism convolution, and weighted channel fusion. This reduces the number of channels in the output feature image, reduces the computational load of the image processing process, and makes reasonable use of the differences in feature representation capabilities of different levels of features. In addition, this image processing method can detect printed text and handwritten text in parallel, which can reduce the erasure of printed text and facilitate the erasure and image restoration of handwritten text.

[0050] It should be noted that this disclosure is an illustrative example using a teaching scenario, and the technical solutions provided by the embodiments of this disclosure can still be applied to other fields involving the need to erase handwritten text and restore images.

[0051] It should be noted that, in this disclosure, processing operations such as convolution, downsampling, and upsampling can be executed or implemented through layers such as convolutional layers, downsampling layers, and upsampling layers, respectively. Correspondingly, these layers can also be used to refer to the corresponding processing operations, which will not be repeated below.

[0052] The following detailed description, with reference to the accompanying drawings, outlines some embodiments and examples of this disclosure. It should be understood that the specific implementations described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure.

[0053] Figure 1 This is a flowchart illustrating an image processing method provided in at least some embodiments of the present disclosure. For example, this image processing method can be applied to a computing device, which includes any electronic device with computing capabilities, such as a smartphone, laptop, tablet, desktop computer, server, etc., and the embodiments of the present disclosure are not limited thereto. For example, the computing device has a Central Processing Unit (CPU) or a Graphics Processing Unit (GPU), and also includes memory. This memory is, for example, non-volatile memory (e.g., read-only memory, ROM) storing operating system code. For example, the memory also stores code or instructions that, by running these codes or instructions, can implement the image processing method provided in the embodiments of the present disclosure. Figure 2 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of steps S200 and S300 is shown below. The following is a combination of... Figure 2 ,right Figure 1 The image processing method shown will be described in detail.

[0054] For example, such as Figure 1 As shown, the image processing method includes the following steps S100 to S500.

[0055] Step S100: Obtain the input image, wherein the input image includes printed text and handwritten text.

[0056] For example, the input image may include a photograph taken of teaching materials (e.g., books, test papers, and workbooks) using a smartphone camera, tablet camera, or digital camera lens, or an image obtained by scanning the teaching materials (e.g., books, test papers, and workbooks) using a printer or other scanning device. It should be noted that although the embodiments of this disclosure use test paper images as examples of input images, this should not be considered a limitation of this disclosure.

[0057] For example, in some embodiments, the input image can be a color image. For example, a color image includes, but is not limited to, a color image with three color channels. For example, the three color channels include a first color channel, a second color channel, and a third color channel. For example, the three color channels correspond to the three primary colors. For example, in some examples, the first color channel is the red (R) channel, the second color channel is the green (G) channel, and the third color channel is the blue (B) channel; that is, the aforementioned color image can be an RGB format color image. It should be noted that the embodiments of this disclosure include, but are not limited to, this. For example, in other embodiments, the input image can also be a grayscale image. It is understood that in educational scenarios, input images (e.g., photos or pictures of various teaching materials) are primarily grayscale images.

[0058] For example, printed text can include printed marks such as words, lines, and graphics formed by printing, while handwritten text can include handwritten marks such as words, lines, and graphics formed by handwriting.

[0059] Step S200: Based on the input image, obtain N sets of initial feature images arranged from high to low resolution, where N is a positive integer and N≥2.

[0060] For example, a convolutional neural network can be used to process the input image to extract N sets of initial feature images arranged from high to low resolution. It is understood that each of these N sets of initial feature images belongs to a different level (each set of feature images corresponds to one level). For example, feature extraction operations in common text detection algorithms / models (such as Craft, PSENet, PANNet, DBNet, FCENet, etc.) can be referenced to extract multiple sets of feature images (i.e., N sets of initial feature images) at different levels from the input image. It should be noted that embodiments of this disclosure include, but are not limited to, this.

[0061] Figure 3 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of step S200 is shown below. The following is a combination of... Figure 2 and Figure 3 Step S200 is described in detail, but should not be construed as limiting the embodiments of this disclosure.

[0062] For example, such as Figure 3 As shown, based on the input image, N sets of initial feature images are obtained in descending order of resolution, i.e., step S200, which may include the following steps S210 and S220.

[0063] Step S210: Perform analysis processing on the input image M times consecutively to obtain M sets of initial feature images arranged from high to low resolution, where M is a positive integer and M≥N.

[0064] For example, analysis processing can typically include convolution processing, activation processing, and downsampling processing. For example, analysis processing can further include normalization processing. It should be noted that the embodiments disclosed herein are not limiting in this regard.

[0065] Convolutional processing can be implemented through convolutional layers. A convolutional layer applies several convolutional kernels (also called filters) to an input image to extract various types of features. Each kernel extracts one type of feature. Convolutional kernels are typically initialized as random fractional matrices, and during the training of the convolutional neural network, they learn appropriate weights. The result obtained after applying a convolutional kernel to the input image is called a feature image, and the number of feature images is equal to the number of convolutional kernels.

[0066] Activation processing can be implemented through activation layers, which include activation functions. These activation functions introduce nonlinearity into the convolutional neural network (CNN), enabling it to better solve more complex problems. Activation functions can include ReLU (Recursive Luminaire), sigmoid, or tanh functions. ReLU is a non-saturating nonlinear function, while sigmoid and tanh are saturating nonlinear functions. For example, activation layers can be a standalone layer in a convolutional neural network, or they can be included within convolutional layers.

[0067] Downsampling is used to reduce the size (resolution) of a feature image, thereby reducing the amount of data in the feature image. This can be achieved through downsampling layers, but is not limited to these methods. For example, downsampling layers can employ downsampling methods such as max pooling, average pooling, strided convolution, decimation (e.g., selecting a fixed number of pixels), and demuxout (splitting the input image into multiple smaller images). Downsampling layers can also use interpolation algorithms such as interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. For instance, when using interpolation algorithms for downsampling, only the interpolated values ​​can be retained while the original pixel values ​​are removed, thus reducing the size of the feature image.

[0068] Normalization can be achieved through a normalization layer, which allows the pixel values ​​of the feature image to vary within a predetermined range, thereby simplifying the feature image generation process and improving the image processing effect. For example, the predetermined range can be [-1, 1], etc.

[0069] It is understandable that the M initial feature images obtained in step S210 form a feature pyramid.

[0070] Step S220: Select N sets of initial feature images from the M sets of initial feature images (i.e., N sets of initial feature images arranged from high to low resolution).

[0071] For example, when M=N, the selection process in step S220 can be omitted, that is, N sets of initial feature images arranged from high to low resolution can be obtained directly through step S210.

[0072] For example, when M>N, the N sets of initial feature images typically include the last set of initial feature images (i.e., the smallest or lowest-resolution initial feature image) from the M sets of initial feature images obtained in step S210, and N-1 sets of initial feature images selected from the first M-1 sets of initial feature images of the M sets of initial feature images. It should be noted that the embodiments of this disclosure do not limit the selection method. It is understood that the N sets of initial feature images used for subsequent processing also form a feature pyramid.

[0073] For example, in some embodiments, in step S200, the number of layers of the N initial feature images used for subsequent processing can be set to avoid increasing unnecessary computation (i.e., avoiding computational redundancy). For example, N≤4; or N=3. M can be equal to N or slightly greater than N (e.g., M=N+1, but not limited thereto).

[0074] For example, in some embodiments, in step S210, the downsampling factor of each analysis process in the M analysis processes (i.e., the downsampling factor of each analysis process) can be set to 2. Meanwhile, to avoid the size of the last initial feature image in the M initial feature images (i.e., the last initial feature image in the N initial feature images) being too small, the size of M (and N) can be set. For example, N ≤ 4; or N = 3. M can be equal to N or slightly greater than N (e.g., M = N + 1, but not limited thereto).

[0075] For example, in educational scenarios, the text in teaching materials such as test papers is relatively small. If M (or N) is too large, the overall downsampling factor (relative to the input image) of the last set of initial feature images will be too large, resulting in the size of the last set of initial feature images being too small, which may lead to a large area of ​​text being missed. Therefore, it is usually necessary to set the size of M (or N) and the downsampling factor for each analysis process reasonably. For example, in a specific example, such as Figure 2 As shown, M=4, N=3, and the downsampling factor for each analysis is 2. Specifically, as... Figure 2As shown, the input image can be analyzed four times consecutively to obtain four sets of initial feature images F0 to F3 arranged from high to low resolution. Specifically, the input image is analyzed to obtain the 0th set of initial feature images F0, whose resolution is half that of the input image. The 0th set of initial feature images F0 is then analyzed to obtain the 1st set of initial feature images F1, whose resolution is half that of the 0th set of initial feature images F0. The resolution of the first initial feature image F1 is 1 / 4 of the original image resolution. The first initial feature image F1 is analyzed to obtain the second initial feature image F2, whose resolution is half that of the first initial feature image F1, i.e., 1 / 8 of the original image resolution. The second initial feature image F2 is then analyzed to obtain the third initial feature image F3, whose resolution is half that of the second initial feature image F2, i.e., 1 / 16 of the original image resolution. Then, the last three initial feature images (F1, F2, and F3) can be selected from the four initial feature images F0 to F3 for subsequent processing. That is, in N initial feature images, the resolution of the i-th initial feature image can be twice the resolution of the (i+1)-th initial feature image, where i = 1, 2, ..., N-1. In this context, the image processing method provided by the embodiments of this disclosure has high adaptability in educational scenarios.

[0076] Step S300: Based on N sets of initial feature images, obtain the output feature image.

[0077] For example, operations such as convolution, upsampling, channel self-attention mechanism convolution and weighted channel fusion can be combined to process the N initial feature images obtained in step S200 to obtain output feature images. This can reduce the number of channels in the output feature images, reduce the computational load of the image processing process, and reduce the difference in feature representation ability between different levels of features.

[0078] Figure 4 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of step S300 is shown below. The following is a combination of... Figure 2 and Figure 4 Step S300 is described in detail, but should not be construed as limiting the embodiments of this disclosure.

[0079] For example, such as Figure 4As shown, based on N sets of initial feature images, an output feature image is obtained, i.e., step S300, which may include steps S310 to S350. It should be understood that in the following description, the order of each set of initial feature images is relative to the N sets of initial feature images.

[0080] Step S310: Upsample the (i+1)th initial feature image to obtain the i-th upsampled feature image with the same resolution as the i-th initial feature image, and perform channel concatenation and convolution on the i-th initial feature image and the i-th upsampled feature image to obtain the i-th intermediate feature image, where i = 1, 2, ..., N-1.

[0081] For example, the resolution of the i-th intermediate feature image can be the same as the resolution of the i-th initial feature image, where i = 1, 2, ..., N-1. It should be noted that embodiments of this disclosure include, but are not limited to, this.

[0082] Upsampling is used to increase the size of a feature image, thereby increasing the amount of data in the feature image. This can be achieved through upsampling layers, but is not limited to these methods. For example, upsampling layers can employ methods such as strided transposed convolution and interpolation algorithms. Interpolation algorithms can include, for example, interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. For instance, when using interpolation algorithms for upsampling, both the original pixel values ​​and the interpolated values ​​can be preserved, thus increasing the size of the feature image.

[0083] Channel stitching is used to stack the channels of multiple (e.g., two or more) images to be stitched together, so that the number of channels of the stitched image is the sum of the number of channels of the multiple images to be stitched together.

[0084] For example, in a specific example, such as Figure 2As shown, the second set of initial feature images F2 is upsampled to obtain a first set of upsampled feature images with the same resolution as the first set of initial feature images F1. The first set of initial feature images F1 and the first set of upsampled feature images are then concatenated and convolved to obtain a first set of intermediate feature images P1. Similarly, the third set of initial feature images F3 is upsampled to obtain a second set of upsampled feature images with the same resolution as the second set of initial feature images F2. The second set of initial feature images F2 and the second set of upsampled feature images are then concatenated and convolved to obtain a second set of intermediate feature images P2. For example, the resolution of the first set of intermediate feature images P1 is the same as the resolution of the first set of initial feature images F1, and the resolution of the second set of intermediate feature images P2 is the same as the resolution of the second set of initial feature images F2. It is understood that since the downsampling factor is set to 2 for each analysis, the upsampling factor for this upsampling process is also set to 2.

[0085] Step S320: Perform convolution processing on the Nth group of initial feature images to obtain the Nth group of intermediate feature images.

[0086] For example, the resolution of the Nth group of intermediate feature images can be the same as the resolution of the Nth group of initial feature images. It should be noted that embodiments of this disclosure include, but are not limited to, this.

[0087] For example, in a specific example, such as Figure 2 As shown, the resolution of the third group of intermediate feature images P3 is the same as the resolution of the third group of initial feature images F3.

[0088] Step S330: Perform convolution processing of the Nth initial feature image using the channel self-attention mechanism to obtain the N+1th intermediate feature image.

[0089] For example, self-attention is a variant of attention mechanisms that reduces reliance on external information and is better at capturing the internal correlations of data or features. For instance, the implementation process and details of self-attention mechanisms can be found in descriptions of models such as the Transformer in the fields of artificial intelligence or artificial neural networks, and will not be repeated here. For example, the convolutional processing of channel self-attention mechanisms can be implemented using 1×1 convolutional kernels; it should be noted that embodiments of this disclosure include, but are not limited to, this.

[0090] For example, in a specific example, such as Figure 2 As shown, the resolution of the fourth group of intermediate feature images P4 is the same as the resolution of the third group of initial feature images F3. It should be noted that embodiments of this disclosure include, but are not limited to, this.

[0091] Step S340: Perform convolution processing on the first group of intermediate feature images to obtain the first group of transition feature images. Perform convolution processing and upsampling processing on the j-th group of intermediate feature images to obtain the j-th group of transition feature images, where j = 2, 3, ..., N+1, and the resolution and number of channels of the N+1 groups of transition feature images are the same.

[0092] For example, the resolution of all N+1 groups of transition feature images can be the same as the resolution of the first group of intermediate feature images. It should be noted that the embodiments of this disclosure include, but are not limited to, this.

[0093] For example, in a specific example, such as Figure 2 As shown, the first set of intermediate feature images P1 undergoes convolution processing to obtain the first set of transition feature images; the second set of intermediate feature images P2 undergoes convolution processing and upsampling processing (2x upsampling) to obtain the second set of transition feature images; the third set of intermediate feature images P3 undergoes convolution processing and upsampling processing (4x upsampling) to obtain the third set of transition feature images; and the fourth set of intermediate feature images P4 undergoes convolution processing and upsampling processing (4x upsampling) to obtain the fourth set of transition feature images. It can be understood that in step S340, when processing each set of intermediate feature images, the number of convolution kernels for the corresponding convolution processing and the upsampling factor for the upsampling processing can be reasonably set to ensure that the number of channels and resolution of each set of transition feature images are the same.

[0094] Step S350: Perform weighted channel fusion processing on the N+1 groups of transition feature images to obtain the output feature image.

[0095] For example, in some embodiments, weighted channel fusion processing is performed on N+1 groups of transition feature images to obtain an output feature image. Specifically, step S350 may include: performing a pixel-by-pixel, channel-by-channel weighted summation operation on the N+1 groups of transition feature images based on an attention layer to obtain the output feature image; wherein the attention layer includes N+1 nodes, each node indicating the weights used in the weighted summation operation on the N+1 groups of transition feature images. Thus, the output feature image fuses features from different levels.

[0096] For example, the design and details of the attention layer can be found in the descriptions of attention mechanism models in the fields of artificial intelligence or artificial neural networks, and will not be repeated here. It can be understood that the attention mechanism filters out a small amount of important information from a large amount of data and focuses on this important information, while ignoring most of the unimportant information. The larger the weight of a node, the more it focuses on its corresponding set of transition feature images. The weight represents the importance of the information, and the corresponding set of transition feature images is its corresponding information. Therefore, by performing a pixel-by-pixel, channel-by-channel weighted summation operation on N+1 sets of transition feature images based on the attention layer to obtain the output feature image, the differences in feature representation capabilities of different levels of features can be reasonably utilized.

[0097] Step S400: Based on the output feature image, perform dual-class text detection processing to obtain printed text detection mask image and handwritten text detection mask image.

[0098] For example, a convolutional neural network capable of outputting dual-channel feature images can be used to perform dual-class text detection processing on the output feature images to obtain printed text detection mask images and handwritten text detection mask images. For instance, the dual-class text detection processing here can refer to the text detection operations in common text detection algorithms / models (such as Craft, PSENet, PANNet, DBNet, FCENet, etc.), simply by modifying the text detection operation from single-channel output to dual-channel output. It should be noted that embodiments of this disclosure include, but are not limited to, this.

[0099] For example, the resolution of the printed text detection mask image is the same as the resolution of the handwritten text detection mask image. For example, in some embodiments, the printed text detection mask image and the handwritten text detection mask image have the same resolution as the input image; for example, in other embodiments, the printed text detection mask image and the handwritten text detection mask image obtained directly from the dual-class text detection processing have different resolutions than the input image. In this case, the printed text detection mask image and the handwritten text detection mask image obtained directly from the dual-class text detection processing can be scaled, and the scaled printed text detection mask image and the handwritten text detection mask image can be used as the printed text detection mask image and the handwritten text detection mask image obtained in step S400.

[0100] It should be noted that before processing the input image using the image processing method provided in the embodiments of this disclosure, it is usually necessary to train the neural network structure (such as convolutional layers) involved in steps S200 to S400. The training process can refer to common training methods, which will not be elaborated here. For example, in some embodiments, for printed text detection mask images and handwritten text detection mask images, a channel-independent logistic classification cross-entropy loss function can be designed to calculate the network loss and perform gradient backpropagation to update and optimize the network parameters. For example, the logistic classification cross-entropy loss function is as follows:

[0101]

[0102] Where L represents the network loss, M represents the number of input image samples in a training batch, H and W represent the width and height of the printed text detection mask image / handwritten text detection mask image, and x represents the height of the input image. mik σ(x) represents the pixel value of the k-th output channel corresponding to the i-th pixel position of the m-th input image sample. mik This indicates that the Sigmoid function is used on x. mik The result obtained after processing, y mik w represents the true label of the k-th output channel corresponding to the i-th pixel position of the m-th input image sample. k This represents the weight corresponding to the k-th output channel. k is the index of the output channel; for example, k=1 corresponds to the printed text detection mask image, k=2 corresponds to the handwritten text detection mask image, or vice versa.

[0103] In practical applications, handwritten and printed text sometimes overlap. Directly using the cross-entropy loss function between channels can cause the network to predict only a single category at a given pixel location. In contrast, the loss function used in this embodiment effectively avoids this problem, allowing the network to predict two categories at the same location while ensuring good prediction results for each category. It is understood that in the image processing method provided by the embodiments of this disclosure, using a neural network structure to perform related steps can improve the robustness of the image processing method to text morphology, such as in teaching materials.

[0104] Figure 5 This is a schematic diagram illustrating an input image (e.g., an exam paper image) and its corresponding printed text detection mask image and handwritten text detection mask image, provided for at least some embodiments of this disclosure. Figure 5As shown, based on the operations of steps S200 to S400 described above, printed text and handwritten text in the input image are accurately detected. In the printed text detection mask image, brighter areas indicate a greater likelihood that they belong to the foreground of printed text, while darker areas indicate a greater likelihood that they belong to the background of printed text. In the handwritten text detection mask image, brighter areas indicate a greater likelihood that they belong to the foreground of handwritten text, while darker areas indicate a greater likelihood that they belong to the background of handwritten text.

[0105] Step S500: Based on the printed text detection mask image and the handwritten text detection mask image, a first output image is obtained, wherein, compared with the input image, the first output image is an image in which at least part of the handwritten text has been erased.

[0106] For example, in step S500, the pixel-level coordinates of the handwritten text can be obtained without losing the printed text information, and then the handwritten text can be erased.

[0107] Figure 6 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 1 An exemplary flowchart of step S500 is shown below. The following is a combination of... Figure 6 Step S500 is described in detail, but should not be construed as a limitation on the embodiments of this disclosure.

[0108] For example, such as Figure 6 As shown, the first output image is obtained based on the printed text detection mask image and the handwritten text detection mask image, i.e., step S500, which may include the following steps S510 to S540.

[0109] Step S510: Perform first binarization processing on the printed text detection mask image and the handwritten text detection mask image respectively to obtain the printed text binarized image and the handwritten text binarized image.

[0110] For example, in some embodiments, the first binarization process may include: step S511, analyzing the pixel value distribution of the input of the first binarization process to determine a first threshold; and step S512, determining the pixel values ​​of the output of the first binarization process according to the relationship between the pixel values ​​of the input of the first binarization process and the first threshold. For example, when the input of the first binarization process is a printed text detection mask image, the output of the first binarization process is a printed text binarized image; when the input of the first binarization process is a handwritten text detection mask image, the output of the first binarization process is a handwritten text binarized image.

[0111] For example, in step S511, the average pixel values ​​of the input from the first binarization process can be used as the first threshold. It should be noted that embodiments of this disclosure include, but are not limited to, this. For example, in practical applications, any other feasible and suitable method (including but not limited to the bimodal method, P-parameter method, Otsu's method, KSW entropy method, etc.) can be used to determine the first threshold. For example, the first threshold can also be manually adjusted based on the pixel value distribution of the input from the first binarization process. It is understood that the first threshold corresponding to the printed text detection mask image and the first threshold corresponding to the handwritten text detection mask image can be the same or different.

[0112] For example, in step S512, in response to any pixel value of the input to the first binarization process being not less than a first threshold, the corresponding pixel value of the output of the first binarization process can be set to 1; and in response to any pixel value of the input to the first binarization process being less than the first threshold, the corresponding pixel value of the output of the first binarization process can be set to 0. Thus, a binarized image of printed text and a binarized image of handwritten text can be obtained.

[0113] Step S520: Calculate the difference between the binary image of the printed text and the binary image of the handwritten text to obtain the first mask image.

[0114] For example, in some embodiments, a pixel-by-pixel subtraction operation can be performed between the binary image of printed text and the binary image of handwritten text to obtain a first mask image. For example, the binary image of printed text is denoted as M. p The binarized image of handwritten text is denoted as M. h And perform matrix subtraction M r =M p -M h To obtain the first mask image M r It is understandable that the set of values ​​for each pixel in the first mask image is {-1, 0, 1}.

[0115] In real-world teaching scenarios, there may be overlaps between handwritten and printed text. In such cases, directly erasing the handwritten text could lead to the accidental erasure of the printed text. In the first mask image, the area to be erased (i.e., the area containing handwritten text that does not overlap with printed text) corresponds to a pixel-level area with a value of -1, while the overlapping area corresponds to a pixel-level area with a value of 0. This allows for the differentiation between the area to be erased and the overlapping area, thus preventing or mitigating the problem of accidental erasure of printed text in subsequent processing.

[0116] Step S530: Perform a second binarization process on the first mask image to obtain a mask image of the area to be erased.

[0117] For example, in some embodiments, the second binarization process may include: step S531, in response to any pixel value of the first mask image being not less than 0, setting the corresponding pixel value of the mask image of the region to be erased to 1; and step S532, in response to any pixel value of the first mask image being less than 0, setting the corresponding pixel value of the mask image of the region to be erased to 0. Thus, in the mask image of the region to be erased, the region to be erased corresponds to a pixel-level region with a pixel value of 0, and the remaining regions correspond to pixel-level regions with a pixel value of 1.

[0118] It is understandable that the first binarization process in step S510 and the second binarization process in step S530 are essentially both binarization processes, with only slight differences in processing details (e.g., threshold differences). Therefore, "first" and "second" are only used to distinguish them.

[0119] Step S540: Perform pixel-by-pixel multiplication on the input image and the mask image of the area to be erased to obtain the first output image.

[0120] For example, the mask image of the area to be erased has the same resolution as the input image, thus enabling pixel-by-pixel multiplication. It is understood that the resolution of the mask image of the area to be erased is the same as the resolution of the printed text detection mask image and the handwritten text detection mask image; if the resolutions of the printed text detection mask image and the handwritten text detection mask image used in step S500 are different from the resolution of the input image, then the mask image of the area to be erased can be scaled before being multiplied pixel-by-pixel with the input image. It is understood that in the first output image, handwritten text that does not overlap with the printed text is erased.

[0121] Figure 7 This is a schematic diagram illustrating an input image and its corresponding printed text detection mask image, handwritten text detection mask image, and a first output image, provided for at least some embodiments of this disclosure. For example... Figure 7 As shown, compared to the input image, at least a portion of the handwritten text has been erased in the first output image, wherein the pixel value of the area where the at least portion of the handwritten text is located is 0 (e.g., the visual effect is a black overlay).

[0122] Understandable Figure 1 The image processing method shown can effectively erase handwritten text and restore the image to a certain extent. However, the teaching materials printed from the first output image have a black overlay effect. This black overlay effect usually does not affect the recognition and use of the teaching materials, but it may still be detrimental to writing. Therefore, the first output image can be further processed to obtain an image with a more aesthetically pleasing local background that is easier to write on.

[0123] Figure 8 A flowchart illustrating another image processing method provided for at least some embodiments of this disclosure. For example, such as... Figure 8 As shown, in Figure 1 Based on the image processing method shown (i.e., steps S100 to S500), the image processing method may include the following steps S600 to S700.

[0124] Step S600: Perform a simulated background replacement process on the first output image to obtain the second output image.

[0125] For example, in the first output image, the pixels of the erased handwritten text area have a black overlay effect. In this case, the overall visual effect of the first output image may not be aesthetically pleasing. To restore the original paper background of the erased handwritten text area, a simulated background replacement process can be performed on the first output image to obtain the second output image.

[0126] Figure 9 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 8 An exemplary flowchart of step S600 is shown below. The following is a combination of... Figure 9 Step S600 is described in detail, but should not be construed as a limitation on the embodiments of this disclosure.

[0127] For example, such as Figure 9 As shown, the first output image is subjected to a simulated background replacement process to obtain the second output image, i.e., step S600, which may include the following steps S610 to S640.

[0128] Step S610: Blur the input image to obtain a simulated background image.

[0129] For example, in some embodiments, a Gaussian blur algorithm can be used to blur the input image to obtain a simulated background image. For example, details of the Gaussian blur algorithm can be found in the description of the Gaussian blur operation (GuassianBlur function) in OpenCV (an open-source computer vision library), and will not be repeated here.

[0130] For example, the blur radius of the Gaussian blur algorithm can be set to min(w,h,R), where w and h represent the width and height of the input image, respectively, and R is a preset value, where w, h, and R are all positive integers. For example, a Gaussian blur filter with a large convolutional kernel can be used to process the input image to obtain a simulated background image robust to background lighting and original background information. For example, in some embodiments, R ≥ 100, and embodiments of this disclosure include, but are not limited to, this. For example, in a specific example, R = 299. It should be noted that in practical applications, the preset value R can be set according to actual needs.

[0131] Step S620: Perform inverse binarization on the mask image of the area to be erased to obtain a second mask image.

[0132] For example, the area to be erased in step S620 is the mask image of the region to be erased obtained in step S530. For example, the reverse binarization process may include: if the original pixel value (i.e., the pixel value in the mask image of the region to be erased) is 1, then change it to 0 (i.e., set the corresponding pixel value in the second mask image to 0); if the original pixel value (i.e., the pixel value in the mask image of the region to be erased) is 0, then change it to 1 (i.e., set the corresponding pixel value in the second mask image to 1). That is, in the second mask image, the region to be erased corresponds to a pixel-level region with a pixel value of 1, and the remaining regions correspond to pixel-level regions with a pixel value of 0.

[0133] Step S630: Perform pixel-by-pixel multiplication on the simulated background image and the second mask image to obtain the background template image.

[0134] For example, the simulated background image and the second mask image have the same resolution, thus enabling pixel-by-pixel multiplication. It is understood that in the background template image, only the pixel values ​​of the area to be erased are non-zero, and the visual effect of this area is close to that of the original paper background.

[0135] Step S640: Perform pixel-by-pixel addition on the first output image and the background template image to obtain the second output image.

[0136] For example, the first output image has the same resolution as the background template image, thus enabling pixel-by-pixel addition.

[0137] It is understandable that, considering the operations in steps S610 to S640, in order to reduce the amount of computation, in step S610, only the region in the input image corresponding to the area to be erased can be blurred without affecting the visual effect of the second output image; that is, in step S610, it is not necessary to blur the entire input image.

[0138] Figure 10 This is a schematic diagram illustrating an input image and its corresponding first output image, simulated background image, and second output image, provided for at least some embodiments of this disclosure. For example... Figure 10 As shown, compared to the first output image, the second output image is equivalent to replacing the black-covered erased area with a state close to the original paper background (e.g., with a shadowed visual effect). In other words, step S600 can effectively mimic the paper background of the erased area of ​​handwritten text, which is beneficial for further image restoration.

[0139] It should be noted that in the second output image, there is still a slight difference in color between the erased area and the original paper background color that is perceptible to the human eye, which is not aesthetically pleasing. Therefore, the second output image can be further processed to obtain an image that is closer to the original image (i.e., an image with only printed text and no handwritten text, such as an image of teaching materials before they are written on).

[0140] Step S700: Perform adaptive light and shadow removal processing on the second output image to obtain the third output image.

[0141] For example, in the second output image, the pixels in the erased area of ​​the handwritten text have a shadow covering effect. In this case, the overall visual effect of the second output image may not be aesthetically pleasing. To improve the overall aesthetics, adaptive shadow removal processing can be applied to the second output image to obtain the third output image.

[0142] Figure 11 A corresponding embodiment provided for at least some embodiments of this disclosure Figure 8 An exemplary flowchart of step S700 is shown below. The following is a combination of... Figure 11 Step S700 is described in detail, but should not be construed as a limitation on the embodiments of this disclosure.

[0143] For example, such as Figure 11 As shown, adaptive light and shadow removal processing is performed on the second output image to obtain the third output image, i.e., step S700, which may include the following steps S710 to S730.

[0144] Step S710: Perform adaptive threshold binarization on the second output image to obtain the third mask image.

[0145] For example, in some embodiments, the adaptive threshold binarization process in step S710 may include: step S711, in response to any pixel value in the second output image being not less than the weighted sum of pixel values ​​of its neighboring blocks, setting the corresponding pixel value of the third mask image to 1; and step S712, in response to any pixel value in the second output image being less than the weighted sum of pixel values ​​of its neighboring blocks, setting the corresponding pixel value of the third mask image to 0. That is, in the adaptive threshold binarization process in step S710, the weighted sum of pixel values ​​of the neighboring blocks of each pixel value is used as its adaptive threshold parameter. For example, the aforementioned neighborhood block can be a Gaussian neighborhood block, and embodiments of this disclosure include, but are not limited to, this. For example, the size of the Gaussian neighborhood block can be set according to actual needs; for example, in a specific example, the size of the Gaussian neighborhood block can be set to, for example, 19. It should be noted that embodiments of this disclosure do not limit this.

[0146] It should be noted that in practical applications, other feasible and suitable methods can also be used for adaptive threshold binarization processing, and the embodiments disclosed herein do not limit this.

[0147] Step S720: Smooth the third mask image to obtain an intermediate output image.

[0148] For example, in some embodiments, a closing operation can be used for the smoothing process in step S720. For instance, a closing operation filter kernel can be constructed and used to process the third mask image to obtain an intermediate output image. It is understood that the closing operation filter kernel can be constructed according to actual needs. The closing operation is generally used to smooth target contours, bridge narrow gaps and long, thin grooves, eliminate small holes, and fill breaks in contour lines; it is understood that in the third mask image, the highlighted areas (i.e., areas with a pixel value of 1) represent the target. For details on the closing operation, please refer to the relevant description of the closing operation operation in OpenCV, which will not be repeated here.

[0149] It should be noted that in practical applications, any other feasible and suitable methods can be used for smoothing, and the embodiments disclosed herein do not impose any limitations on this.

[0150] Step S730: Perform image dilation processing on the intermediate output image to obtain the third output image.

[0151] For example, image dilation is applied to the highlighted areas of the intermediate output image, making the text in the image appear fuller and more aesthetically pleasing. Details regarding image dilation can be found in the description of image dilation operations in OpenCV, and will not be repeated here.

[0152] Figure 12This is a schematic diagram illustrating a second output image and a third output image provided for at least some embodiments of this disclosure. For example... Figure 12 As shown, compared with the second output image, the third output image effectively eliminates the visual effect of shadow coverage in the erased area, thus improving the overall aesthetics of the image restoration.

[0153] Figure 13 This is a schematic diagram illustrating an input image and a third output image provided for at least some embodiments of this disclosure. For example... Figure 13 As shown, compared with the input image, the handwritten text in the third output image is erased, realizing image restoration. At the same time, the image shadows caused by ambient light and occlusion are adaptively eliminated, improving the overall aesthetics.

[0154] It should be noted that, in the embodiments of this disclosure, the flow of the image processing method described above may include more or fewer operations, which may be executed sequentially or in parallel. Although the flow of the image processing method described above includes multiple operations appearing in a specific order, it should be clearly understood that the order of the multiple operations is not limited. The image processing method described above may be executed once or multiple times according to predetermined conditions.

[0155] The image processing method provided in the embodiments of this disclosure can obtain an output feature image based on a feature pyramid formed by N initial feature images, combined with operations such as convolution, upsampling, channel self-attention mechanism convolution, and weighted channel fusion. This reduces the number of channels in the output feature image, reduces the computational load of the image processing process, and makes reasonable use of the differences in feature representation capabilities of different levels of features. In addition, this image processing method can detect printed text and handwritten text in parallel, which can reduce the erasure of printed text and facilitate the erasure and image restoration of handwritten text.

[0156] At least some embodiments of this disclosure also provide an image processing apparatus. Figure 14 This is a schematic block diagram of an image processing apparatus provided for at least some embodiments of the present disclosure. For example, such as... Figure 14 As shown, the image processing device 100 includes a memory 110 and a processor 120.

[0157] For example, memory 110 is used to store computer-readable instructions in a non-transitory manner, and processor 120 is used to run the computer-readable instructions, which are executed by processor 120 to perform the image processing method provided in any embodiment of this disclosure.

[0158] For example, memory 110 and processor 120 can communicate with each other directly or indirectly. For example, in some examples, such as... Figure 14As shown, the image processing apparatus 100 may further include a system bus 130, through which the memory 110 and the processor 120 can communicate with each other. For example, the processor 120 can access the memory 110 through the system bus 130. For example, in other examples, components such as the memory 110 and the processor 120 can communicate via a network connection. The network may include a wireless network, a wired network, and / or any combination of wireless and wired networks. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things (IoT) based on the Internet and / or a telecommunications network, and / or any combination of the above networks. Wired networks may use methods such as twisted-pair cables, coaxial cables, or fiber optic transmission for communication, while wireless networks may use methods such as 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee, or WiFi. This disclosure does not limit the type and function of the network.

[0159] For example, processor 120 can control other components in the image processing device to perform desired functions. Processor 120 can be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), tensor processor (TPU), or graphics processing unit (GPU). The CPU can be based on x86 or ARM architectures. The GPU can be integrated directly onto the motherboard or built into the motherboard's northbridge chip. The GPU can also be integrated into the CPU.

[0160] For example, memory 110 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0161] For example, one or more computer instructions may be stored on memory 110, and processor 120 may execute the computer instructions to perform various functions. Various application programs and various data may also be stored in the computer-readable storage medium, such as input images, initial feature images, output feature images, first output images, second output images, intermediate output images, third output images, and various data used and / or generated by application programs.

[0162] For example, some computer instructions stored in memory 110 can be executed by processor 120 to perform one or more steps in the image processing method described above.

[0163] For example, such as Figure 14 As shown, the image processing apparatus 100 may further include an input interface 140 that allows external devices to communicate with the image processing apparatus 100. For example, the input interface 140 may be used to receive instructions or data (e.g., input images, etc.) from external computer devices, users, etc. The image processing apparatus 100 may further include an output interface 150 that enables the image processing apparatus 100 to connect to one or more external devices. For example, the image processing apparatus 100 may output image processing results (e.g., a first output image, a second output image, an intermediate output image, a third output image, etc.) through the output interface 150. External devices that communicate with the image processing apparatus 100 through the input interface 140 and the output interface 150 may be included in an environment that provides any type of user interface that a user can interact with. Examples of user interface types include graphical user interfaces, natural user interfaces, etc. For example, a graphical user interface may accept input from a user using an input device such as a keyboard, mouse, remote control, etc., and provide output on an output device such as a display. Furthermore, a natural user interface allows a user to interact with the image processing apparatus 100 in a manner that is not constrained by input devices such as a keyboard, mouse, remote control, etc. In contrast, natural user interfaces can rely on voice recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and semantics, vision, touch, gestures, and machine intelligence.

[0164] In addition, the image processing device 100, although in Figure 14 While shown as a single system, it is understood that the image processing apparatus 100 can also be a distributed system, and may be deployed as a cloud facility (including a public or private cloud). Thus, for example, several devices can communicate via a network connection and collaboratively perform tasks described as being performed by the image processing apparatus 100. For example, in some embodiments, an input image can be acquired by a client and uploaded to a server; the server performs image processing based on the received input image and then returns, for example, a third output image to the client for use by the user.

[0165] For example, a detailed description of the implementation process of the image processing method can be found in the relevant descriptions in the embodiments of the above image processing method, and repeated details will not be repeated here.

[0166] For example, in some cases, the image processing device may include, but is not limited to, smartphones, tablets, personal computers, personal digital assistants (PDAs), servers, etc.

[0167] It should be noted that the image processing apparatus provided in the embodiments of this disclosure is exemplary and not restrictive. Depending on the actual application needs, the image processing apparatus may also include other conventional components or structures. For example, in order to realize the necessary functions of the image processing apparatus, those skilled in the art may set other conventional components or structures according to specific application scenarios. The embodiments of this disclosure do not limit this.

[0168] The technical effects of the image processing apparatus provided in the embodiments of this disclosure can be referred to the corresponding descriptions of the image processing methods in the above embodiments, and will not be repeated here.

[0169] At least some embodiments of this disclosure also provide a non-transitory storage medium. Figure 15 This is a schematic diagram of a non-transitory storage medium provided as an embodiment of the present disclosure. For example, such as Figure 15 As shown, the non-transitory storage medium 200 stores computer-readable instructions 201 non-transitoryly. When the non-transitory computer-readable instructions 201 are executed by a computer (including a processor), the image processing method provided in any embodiment of this disclosure can be executed.

[0170] For example, one or more computer instructions may be stored on the non-transitory storage medium 200. Some of the computer instructions stored on the non-transitory storage medium 200 may be, for example, instructions for implementing one or more steps in the image processing method described above.

[0171] For example, non-transitory storage media may include storage components of a tablet computer, hard disks of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0172] The technical effects of the non-transient storage medium provided in the embodiments of this disclosure can be found in the corresponding descriptions of the image processing methods in the above embodiments, and will not be repeated here.

[0173] The following points need to be clarified regarding this disclosure:

[0174] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0175] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0176] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An image processing method, comprising: Obtain an input image, wherein the input image includes printed text and handwritten text; Based on the input image, N initial feature images are obtained, arranged from high to low resolution, where N is a positive integer and N ≥ 2; Based on the N sets of initial feature images, the output feature image is obtained; Based on the output feature image, dual-class text detection processing is performed to obtain a printed text detection mask image and a handwritten text detection mask image; and Based on the printed text detection mask image and the handwritten text detection mask image, a first output image is obtained, wherein, compared with the input image, the first output image is an image in which at least part of the handwritten text has been erased; The output feature image is obtained based on the N sets of initial feature images, including: Upsampling is performed on the (i+1)th initial feature image to obtain the i-th upsampled feature image with the same resolution as the i-th initial feature image. Channel concatenation and convolution are then performed on the i-th initial feature image and the i-th upsampled feature image to obtain the i-th intermediate feature image, where i = 1, 2, …, N-1. The Nth initial feature image is convolved to obtain the Nth intermediate feature image; The Nth initial feature image is subjected to convolution processing using a channel self-attention mechanism to obtain the (N+1)th intermediate feature image; The first set of intermediate feature images is convolved to obtain the first set of transition feature images. The j-th set of intermediate feature images is then convolved and upsampled to obtain the j-th set of transition feature images, where j = 2, 3, …, N+1, and the N+1 sets of transition feature images have the same resolution and number of channels. The N+1 groups of transition feature images are subjected to weighted channel fusion processing to obtain the output feature image.

2. The image processing method according to claim 1, wherein, The weighted channel fusion process is performed on the N+1 groups of transition feature images to obtain the output feature image, including: The output feature image is obtained by performing a pixel-by-pixel, channel-by-channel weighted summation operation on the N+1 groups of transition feature images based on the attention layer. The attention layer includes N+1 nodes, which are used to indicate the weights of the N+1 groups of transition feature images when performing the weighted summation operation.

3. The image processing method according to claim 1 or 2, wherein, N ≤ 4。 4. The image processing method according to claim 3, wherein, N = 3。 5. The image processing method according to claim 1 or 2, wherein, The resolution of the i-th initial feature image is twice the resolution of the (i+1)-th initial feature image, where i = 1, 2, …, N-1.

6. The image processing method according to claim 1 or 2, wherein, Based on the input image, N sets of initial feature images are obtained, arranged from high to low resolution, including: The input image is subjected to M consecutive analysis processes to obtain M sets of initial feature images arranged from high to low resolution, where M is a positive integer and M ≥ N; and Select the N initial feature images from the M initial feature images.

7. The image processing method according to claim 1 or 2, wherein, The printed text detection mask image, the handwritten text detection mask image, the first output image, and the input image have the same resolution.

8. The image processing method according to claim 1 or 2, wherein, Based on the printed text detection mask image and the handwritten text detection mask image, the first output image is obtained, including: The printed text detection mask image and the handwritten text detection mask image are respectively subjected to a first binarization process to obtain the printed text binarized image and the handwritten text binarized image. Calculate the difference between the binary image of the printed text and the binary image of the handwritten text to obtain the first mask image; The first mask image is subjected to a second binarization process to obtain a mask image of the area to be erased; and The input image and the mask image of the area to be erased are multiplied pixel by pixel to obtain the first output image.

9. The image processing method according to claim 8, wherein, The first binarization process includes: Analyze the pixel value distribution of the input to the first binarization process to determine the first threshold; and Based on the relationship between the pixel values ​​of the input of the first binarization process and the first threshold, the pixel values ​​of the output of the first binarization process are determined accordingly. Specifically, when the input to the first binarization process is the printed text detection mask image, the output of the first binarization process is the printed text binarized image; when the input to the first binarization process is the handwritten text detection mask image, the output of the first binarization process is the handwritten text binarized image.

10. The image processing method according to claim 9, wherein, The second binarization process includes: In response to any pixel value of the first mask image being not less than 0, the corresponding pixel value of the mask image of the area to be erased is set to 1; and In response to any pixel value of the first mask image being less than 0, the corresponding pixel value of the mask image of the area to be erased is set to 0.

11. The image processing method according to claim 8, further comprising: The first output image is subjected to a simulated background replacement process to obtain the second output image.

12. The image processing method according to claim 11, wherein, Performing the simulated background replacement process on the first output image to obtain the second output image includes: The input image is blurred to obtain a simulated background image; The mask image of the area to be erased is subjected to inverse binarization to obtain a second mask image; The simulated background image and the second mask image are multiplied pixel by pixel to obtain the background template image; and The first output image and the background template image are added pixel by pixel to obtain the second output image.

13. The image processing method according to claim 12, wherein, The process of blurring the input image to obtain the simulated background image includes: The input image is blurred using a Gaussian blur algorithm to obtain the simulated background image. The blur radius of the Gaussian blur algorithm is set to min(w, h, R), where w and h represent the width and height of the input image, respectively, R is a preset value, w, h, and R are all positive integers, and R ≥ 100.

14. The image processing method according to claim 11, further comprising: The second output image is subjected to adaptive shading removal processing to obtain the third output image.

15. The image processing method according to claim 14, wherein, Performing the adaptive shading removal process on the second output image to obtain the third output image includes: The second output image is subjected to adaptive threshold binarization to obtain the third mask image; The third mask image is smoothed to obtain an intermediate output image; and The intermediate output image is subjected to image dilation processing to obtain the third output image.

16. The image processing method according to claim 15, wherein, The adaptive threshold binarization process is performed on the second output image to obtain the third mask image, including: In response to any pixel value in the second output image being not less than the weighted sum of the pixel values ​​of its neighboring blocks, the corresponding pixel value of the third mask image is set to 1; and In response to any pixel value in the second output image being less than the weighted sum of the pixel values ​​of its neighboring blocks, the corresponding pixel value of the third mask image is set to 0.

17. The image processing method according to claim 15, wherein, The smoothing process applied to the third mask image to obtain the intermediate output image includes: The smoothing process is performed using a closed operation.

18. An image processing apparatus, comprising: Memory is used for non-transitory storage of computer-readable instructions; as well as A processor for executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, perform the image processing method according to any one of claims 1-17.

19. A non-transitory storage medium for non-transitory storage of computer-readable instructions, wherein, When the computer-readable instructions are executed by a computer, the image processing method according to any one of claims 1-17 can be performed.

Images