A Chinese stele text image denoising model construction method, denoising method and device

Abstract

The application discloses a Chinese stele text image denoising model construction method, a denoising method and a device. The model construction method comprises the following steps: collecting multiple complete noisy stele text images to obtain a noise data set; preprocessing the noise data set; taking the preprocessed noise image in the text detection data set as input, taking the corresponding noise image after frame selection as reference output, training an image text detection model, and obtaining a trained image text detection model; frame-out denoising; training a frame-in denoising model to obtain a trained frame-in denoising model, and finally obtaining a Chinese stele text image denoising model. The application can not only effectively remove the speckle noise of the Chinese stele text image, but also effectively remove the scratch noise and sheet corrosion noise which cannot be removed by the existing noise removal model. The application removes all kinds of noises without distinction, reduces the interference of complex noise on the model, and thus realizes the efficiency of the model in removing the noise of the stele text image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image processing, in particular to a Chinese stele text image denoising model construction method and device based on text detection. BACKGROUND

[0002] As an essential part of image processing, the removal of image noise is very important for the image denoising of ancient calligraphy stele text. Due to long-term natural erosion and damage during transportation, spots and cracks are generated on the surface of the stele text. Direct image acquisition of the stele text with spots and cracks will affect the clarity of the stele text image. Therefore, how to remove various noises on the stele text while preserving the original calligraphy details is of great significance for the inheritance and development of calligraphy and has important value for the digital protection and inheritance of cultural heritage.

[0003] Ancient calligraphy stele text not only represents ancient calligraphy art, but also represents Chinese ancient history and culture. In order to better protect and inherit Chinese history and culture, more and more people are committed to the work of cultural relic restoration, and try to restore the original appearance as much as possible. Denoising is the first step and main link of virtual restoration of cultural relics, and the effect of image denoising directly affects the subsequent image processing. After denoising, high-quality stele text images are obtained.

[0004] Existing denoising methods are diverse, but most of them are aimed at removing certain types of noise such as Gaussian noise contained in images. These methods are not suitable for removing noise from stele text images. The noise in stele text images is different from that in general images, and the noise form is not fixed, has no specific distribution rule, and the color is close to the text. The noise form in stele text images can be roughly divided into three categories, namely spot noise, scratch noise and sheet corrosion noise. Most denoising methods can effectively remove spot noise in stele text images. However, when removing scratches and severe sheet corrosion noise, the noise removal effect is not ideal or too many text details are lost. As the first step and main link of virtual restoration of stele text, the quality of denoising result directly affects the subsequent image restoration. Therefore, the denoising of stele text images is very important. SUMMARY

[0005] The present application aims to provide a Chinese stele text image denoising model construction method and device based on text detection, to solve the problem of unsatisfactory denoising effect of existing image denoising methods and devices on Chinese stele text images.

[0006] In order to achieve the above task, the present application adopts the following technical solutions:

[0007] The application provides a Chinese stele text image denoising model construction method, and specifically comprises the following steps:

[0008] Step 1: Collect multiple complete noisy stele text images to obtain a noise data set;

[0009] Step 2: Preprocess the noise data set to obtain a preprocessed noise data set, artificially process the preprocessed data set to obtain a clean data set, and perform text frame selection processing on the clean data set to obtain a corresponding text detection data set;

[0010] Step 3: Take the preprocessed noise image in the text detection data set obtained in step 2 as input, take the corresponding frame-selected noise image as reference output, train an image text detection model, obtain a trained image text detection model, input the preprocessed noise image in the text detection data set into the trained image text detection model, obtain a prediction result and perform out-of-frame denoising on the prediction result, take the prediction result as input, take the clean data set obtained in step 2 as reference output, train an in-frame denoising model, obtain a trained in-frame denoising model, and finally obtain a Chinese stele text image denoising model.

[0011] Further, the step 2 comprises the following substeps:

[0012] Step 2.1: uniformly scale all the stele text images in the noise data set to 512*512 pixels to obtain adjusted stele text images;

[0013] Step 2.2: binarize the adjusted stele text images to obtain binarized stele text images;

[0014] Step 2.3: according to the presentation of the stele text, perform morphological operation on the binarized stele text images to obtain repaired stele text images;

[0015] Step 2.4: artificially denoise all the repaired stele text images to obtain corresponding clean data sets, and pair the preprocessed noise data set obtained in step 2.2 to form a stele text paired data set;

[0016] Step 2.5: perform frame selection processing on all the stele text images in the clean data set obtained in step 2.4 to obtain frame-selected noise images, and pair the frame-selected noise images with the preprocessed noise images obtained in step 2.2 to form a text detection data set.

[0017] Further, the first six layers of the image text detection model comprise a convolution layer, an activation function layer, a Batch Norm layer and a pooling layer connected in sequence, and the seventh to tenth layers comprise a convolution layer, a Batch Norm layer, a convolution layer and a Batch Norm layer connected in sequence; the output of the second layer and the output of the ninth layer are connected in the element direction for jump connection transmission, the output of the third layer and the output of the eighth layer, the output of the fourth layer and the output of the seventh layer, and the output of the fifth layer and the output of the sixth layer are respectively connected in the jump connection; the convolution kernel size of the convolution layer in the first to sixth layers is all set to 3*3, the convolution kernel size of the pooling layer is all set to 2*2, and the step is also set to 2; the first convolution kernel size of the seventh to tenth layers is set to 1*1, and the second is set to 3*3.

[0018] Further, the first four layers of the in-frame denoising model all comprise a convolution layer, a convolution layer and a pooling layer connected in sequence, the fifth layer is a convolution layer, a convolution layer and a Batch Norm layer connected in sequence, and the sixth to ninth layers are an up-sampling layer, a convolution layer and a convolution layer connected in sequence, wherein the output of the first layer and the output of the ninth layer are connected in the element direction for jump connection transmission, the output of the second layer and the output of the eighth layer, the output of the third layer and the output of the seventh layer, and the output of the fourth layer and the output of the sixth layer are respectively connected in the jump connection; the convolution kernel size of the convolution layer in the first to fourth layers is all set to 3*3, the convolution kernel size of the pooling layer is all set to 2*2, and the step is also set to 2; the up-sampling layer of the sixth to ninth layers is realized by using the deconvolution of the 2*2 kernel, and the convolution kernel size of the convolution layer is set to 3*3.

[0019] The second aspect of the present application provides a Chinese stele text image denoising method, which is executed according to the following steps:

[0020] Step one, obtaining a stele text image to be denoised;

[0021] Step two, preprocessing the stele text image to be denoised by the method of step 2 in the Chinese stele text image denoising model construction method of the first aspect of the present application to obtain a preprocessed stele paired image set and a text detection data set;

[0022] Step three, inputting the preprocessed stele paired image set to be denoised into the Chinese stele text image denoising model obtained by the Chinese stele text image denoising model construction method of the first aspect of the present application for processing to obtain a corresponding clean stele image.

[0023] The third aspect of the present application provides a Chinese stele text image denoising model construction device, which comprises a stele image acquisition module, a preprocessing module and a model training module connected in sequence;

[0024] The stele image acquisition module is used for acquiring multiple complete stele images with noise to obtain a noise data set;

[0025] The preprocessing module is used for preprocessing the noise data set to obtain a preprocessed noise data set, and obtaining a clean data set by manual processing of the preprocessed data set; the clean data set is subjected to text box selection processing to obtain a corresponding text detection data set;

[0026] The model training module is used for taking the preprocessed noise image in the text detection data set obtained by the preprocessing module as input, taking the corresponding boxed noise image as reference output, training the image text detection model to obtain a trained image text detection model; the preprocessed noise image in the text detection data set is input into the trained image text detection model to obtain a prediction result, which is subjected to out-of-box denoising, and is taken as input, and the clean data set obtained by the preprocessing module is taken as reference output to train the in-box denoising model to obtain a trained in-box denoising model, and finally obtain a Chinese stele image denoising model.

[0027] Further, the preprocessing module is used for performing the following steps:

[0028] Step 2.1: all stele images in the noise data set are uniformly scaled to 512x512 size pixels to obtain adjusted stele images;

[0029] Step 2.2: the adjusted stele images are subjected to binarization to obtain binarized stele images;

[0030] Step 2.3: according to the stele presentation, morphological operation is performed on the binarized stele images to obtain repaired stele images;

[0031] Step 2.4: manual denoising is performed on all the repaired stele images to obtain corresponding clean data sets, which are paired with the preprocessed noise data sets obtained in step 2.2 to form a stele paired data set;

[0032] Step 2.5: all stele images in the clean data set obtained in step 2.4 are subjected to box selection processing to obtain boxed noise images, and the boxed noise images are paired with the preprocessed noise images obtained in step 2.2 to form a text detection data set.

[0033] Further, the first six layers of the image text detection model comprise convolution layers, activation function layers, Batch Norm layers and pooling layers connected in sequence, and the seventh to tenth layers comprise convolution layers, Batch Norm layers, convolution layers and Batch Norm layers connected in sequence; the output of the second layer and the output of the ninth layer are connected in the element direction for jump connection transmission, the output of the third layer and the output of the eighth layer, the output of the fourth layer and the output of the seventh layer, and the output of the fifth layer and the output of the sixth layer are respectively connected in the jump connection; the convolution kernel size of the convolution layers in the first to sixth layers is all set to 3*3, the convolution kernel size of the pooling layers is all set to 2*2, and the step is also set to 2; the first convolution kernel size of the seventh to tenth layers is set to 1*1, and the second is set to 3*3.

[0034] Further, the first four layers of the in-frame denoising model all comprise convolution layers, convolution layers and pooling layers connected in sequence, the fifth layer is a convolution layer, a convolution layer and a Batch Norm layer connected in sequence, and the sixth to ninth layers are an upsampling layer, a convolution layer and a convolution layer connected in sequence, wherein the output of the first layer and the output of the ninth layer are connected in the element direction for jump connection transmission, the output of the second layer and the output of the eighth layer, the output of the third layer and the output of the seventh layer, and the output of the fourth layer and the output of the sixth layer are respectively connected in the jump connection; the convolution kernel size of the convolution layers in the first to fourth layers is all set to 3*3, the convolution kernel size of the pooling layers is all set to 2*2, and the step is also set to 2; the upsampling layer of the sixth to ninth layers is realized by using the deconvolution of the 2*2 kernel, and the convolution kernel size of the convolution layer is set to 3*3.

[0035] In a fourth aspect of the present application, a Chinese stele text image denoising device is provided, comprising a to-be-denoised stele text image acquisition module, a to-be-denoised stele text image preprocessing module and a denoising module connected in sequence;

[0036] The to-be-denoised stele text image acquisition module is used to obtain a to-be-denoised stele text image;

[0037] The to-be-denoised stele text image preprocessing module is used to preprocess the to-be-denoised stele text image by using the preprocessing module in the Chinese stele text image denoising model construction device of the present application, to obtain a set of preprocessed stele image pairs and a text detection data set;

[0038] The denoising module is used to input the set of preprocessed to-be-denoised stele image pairs into the Chinese stele denoising model obtained by the Chinese stele text image denoising model construction device of the present application for processing, to obtain a corresponding clean stele image.

[0039] Compared with the prior art, the present application has the following technical effects:

[0040] 1. The Chinese stele text image denoising model based on the text detection method and device provided by the application can effectively locate the text range by introducing the text detection, and the default text range outside is noise, so that the noise is removed without distinction. The interference of complex noise on the model is reduced, thereby realizing the efficiency of the model in removing the noise of the stele text image

[0041] 2. The Chinese stele text image denoising model based on the text detection method and device provided by the application combines image text detection and denoising, and ensures the font integrity of the target image obtained in the image denoising process.

[0042] In summary, the application can not only effectively remove the speckle noise of the Chinese stele text image, but also effectively remove the scratch noise and sheet corrosion noise which cannot be removed by the existing noise removal model, and all kinds of noise are removed without distinction. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The network architecture of the image text detection model in the model of the application;

[0044] Figure 2 The network architecture of the whole model in the model of the application;

[0045] Figure 3 The original stele text and the processed stele text are shown in pairs, wherein:

[0046] (a) is the stele text image after size unification; (corresponding to step 2.1)

[0047] (b) is the binary processed stele noise image; (corresponding to step 2.2)

[0048] (c) is the clean stele text image after artificial processing; (corresponding to step 2.4)

[0049] Figure 4 The text detection data pair is shown (corresponding to step 2.5), wherein:

[0050] (a) is the projection of the clean stele text image in different directions;

[0051] (b) is the text frame selection of the clean stele text image;

[0052] (c) is the frame selection of the stele noise image in the same position;

[0053] Figure 5 The denoising result of the stele text image with speckle noise is shown, wherein:

[0054] (a) is the stele text image to be denoised;

[0055] (b) is the result of the frame selection obtained by the image text detection model;

[0056] (c) is the result obtained after the out-of-frame denoising;

[0057] (d) is the final denoising result obtained after in-frame denoising;

[0058] Figure 6 The denoising result of the stele text image with scratch noise is shown, wherein:

[0059] (a) is the stele text image to be denoised;

[0060] (b) is the result of the frame selection obtained by the image text detection model;

[0061] (c) is the result obtained after the out-of-frame denoising;

[0062] (d) is the final denoising result obtained after in-frame denoising;

[0063] Figure 7 The denoising result of the stele text image with patchy corrosion noise is shown, wherein:

[0064] (a) is the stele text image to be denoised;

[0065] (b) is the result of the frame selection obtained by the image text detection model;

[0066] (c) is the result obtained after the out-of-frame denoising;

[0067] (d) is the final denoising result obtained after in-frame denoising. DETAILED DESCRIPTION

[0068] The present application will be described in detail below in conjunction with the accompanying drawings and examples. In order for those skilled in the art to better understand the present application. It needs to be specially reminded that in the following description, when the detailed description of the known function and design may weaken the main content of the present application, these descriptions will be ignored here.

[0069] Example 1

[0070] The present embodiment gives a Chinese stele text image denoising model construction method based on text detection. The model constructed by the method is used to convert the noisy stele text image into a clean stele text image. In the present embodiment, the noisy stele text image can contain different morphological noises such as spots, scratches, patchy corrosion, etc.

[0071] The method is executed according to the following steps:

[0072] Step 1, collect multiple complete noisy stele text images to obtain a noise data set;

[0073] Step 2, pre-processing the noise data set to obtain a pre-processed noise data set, and manually processing the pre-processed data set to obtain a clean data set; performing text frame selection processing on the clean data set to obtain a corresponding text detection data set.

[0074] Preferably, the pre-processing includes uniform image size and binarization; since the size of each epitaph image is not completely consistent, in order to facilitate the training of the neural network, the size of the image is unified in this embodiment, and the image is binarized to make the epitaph image features more obvious and clear, and easier for neural network learning; the manual processing includes morphological operation and manual denoising; the text frame selection processing includes frame selection after projection.

[0075] Specifically, step 2 includes the following sub-steps:

[0076] Step 2.1: scale all epitaph images in the noise data set to 512x512 size pixels to obtain adjusted epitaph images;

[0077] Step 2.2: binarize the adjusted epitaph images to obtain binarized epitaph images, i.e. the pre-processed noise data set;

[0078] In order to further improve the efficiency of neural network learning, the calligraphy data is binarized in this embodiment, and the image is converted from (0, 256) pixels to (0, 1) value pixel image, which is convenient for neural network learning;

[0079] Step 2.3: according to the epitaph presentation, perform morphological operation on the binarized epitaph images to obtain repaired epitaph images;

[0080] In order to obtain a pair of data sets for training the denoising module, the noise data set is subjected to morphological operation in this embodiment, which repairs the epitaph to a certain extent, and the corresponding clean data set is obtained by manual processing; in this embodiment, different morphological operation operations are performed on the epitaph images according to the epitaph presentation, such as filling holes in the text inside, and the like. The operation of repairing can be performed according to the situation, such as the erosion of the text being deep enough to make the strokes thin.

[0081] Step 2.4: manually denoising all epitaph images repaired in step 2.3 to obtain a corresponding clean data set, and pairing the pre-processed noise data set obtained in step 2.2 to form an epitaph pair data set;

[0082] In this embodiment, all epitaph images are enlarged to the pixel level, and the noise part is erased pixel by pixel using image processing software;

[0083] Step 2.5: frame selection processing is performed on all the inscriptions in the clean data set obtained in step 2.4 to obtain a frame-selected noise image, and the frame-selected noise image and the pre-processed noise image obtained in step 2.2 are paired to form a character detection data set;

[0084] In order to obtain the character detection data set for training the character detection module, frame selection processing is performed on the noise image. In this embodiment, the frame selection processing is specifically to perform summation in the horizontal direction on the rows and columns of the matrix obtained by converting the inscriptions in the clean data set, to realize text segmentation and frame selection. The positions of the frame selection of the clean image are correspondingly assigned to the paired original noise image, to realize the character frame selection of the noise image.

[0085] Step 3: the pre-processed noise image in the character detection data set obtained in step 2.5 is taken as input, and the corresponding frame-selected noise image is taken as reference output, to train an image character detection model, and obtain a trained image character detection model. The pre-processed noise image in the character detection data set obtained in step 2.5 is input into the trained image character detection model, to obtain a prediction result, perform out-of-frame denoising (the so-called out-of-frame denoising refers to assigning a pixel value of 1 to each point located outside the frame, i.e., displaying as white), and take the result as input, and take the clean data set obtained in step 2.4 as reference output, to train an in-frame denoising model, and obtain a trained in-frame denoising model.

[0086] The trained image character detection model is used to detect the pre-processed inscription noise image to be tested to obtain a frame-selected inscription noise image of a character region. The trained in-frame denoising model is used to perform in-frame denoising on the frame-selected inscription noise image of the character region to obtain a corresponding clean inscription image.

[0087] A Chinese inscription image denoising model is obtained.

[0088] Preferably, the image character detection model includes as shown in Table 1, wherein the first six layers include convolution layers, activation function layers, Batch Norm layers and pooling layers connected in sequence, and the seventh to tenth layers include convolution layers, Batch Norm layers, convolution layers and Batch Norm layers connected in sequence. The connection mode of the image character detection model network is as shown in Table 1. Figure 1 The output of the second layer of the network and the output of the ninth layer are connected in the element direction to pass through, and the output of the third layer of the network and the output of the eighth layer, the output of the fourth layer and the output of the seventh layer, and the output of the fifth layer and the output of the sixth layer are connected in turn.

[0089] Table 1 Network structure of image character detection model

[0090]

[0091] Batch Norm layer is a layer used to accelerate the training speed in deep learning network, which is placed after the convolution layer or the full connection layer, and is used to normalize the data and accelerate the training fitting speed.

[0092] The convolution kernel size of the convolution layer in the first to sixth layers is set to 3x3, and the convolution kernel size of the pooling layer is set to 2x2, and the step is also set to 2. The first convolution kernel size of the seventh to tenth layers is set to 1x1, and the second is set to 3x3.

[0093] The activation function in the activation function layer is the RELU function.

[0094] Preferably, the in-frame denoising model is shown in Table 2, wherein the first four layers each include a convolution layer, a convolution layer and a pooling layer connected in sequence, the fifth layer is a convolution layer, a convolution layer and a Batch Norm layer connected in sequence, and the sixth to ninth layers are an up-sampling layer, a convolution layer and a convolution layer connected in sequence.

[0095] Table 2 Network structure of in-frame denoising model

[0096]

[0097] The output of the first layer of the network and the output of the ninth layer are connected in the element direction for transmission. In turn, the output of the second layer of the network and the output of the eighth layer, the output of the third layer and the output of the seventh layer, and the output of the fourth layer and the output of the sixth layer are respectively connected in the element direction.

[0098] The convolution kernel size of the convolution layer in the first to fourth layers is set to 3x3, and the convolution kernel size of the pooling layer is set to 2x2, and the step is also set to 2. The up-sampling layer of the sixth to ninth layers is realized by using a 2x2 kernel deconvolution, and the convolution kernel size of the convolution layer is set to 3x3.

[0099] Preferably, in step 3, when training the image text detection model, formula I is used as the loss function :

[0100]

[0101] wherein, represents the stele noise image of the frame selected text region predicted by the trained image text detection network, is the frame selected noise image obtained in step 2.5, represents the image pixel point. is a confidence reference matrix, which is set as shown in formula II:

[0102]

[0103] wherein, The number of text boxes in the image of the epitaph text predicted by the network, The number of text boxes obtained from the image of the noise after the frame selection in the text detection data set obtained in step 2.5. The closer the predicted number of text boxes is to the actual number in the noise image after the frame selection in the text detection data set, the higher the confidence is. For the segmented text box, represent the text box region, represent the image pixel points, and the pixel points in the range are all set to , and the rest are set to 1. Using the loss function can effectively constrain the accurate frame selection of the text region, and the generated frame selection region can as accurately as possible cover the text in the epitaph text image.

[0104] Optionally, when training the in-frame denoising model in step 3, formula III is used as the loss function :

[0105]

[0106] First, the softmax loss function involved needs to be introduced , and the calculation process is . Among them, represents the activation value in the feature channel at the pixel point, is the class number, is the approximate maximum function. For with the maximum activation, 1; when other values, 0.

[0107] Then, the loss function is used to punish the deviation of from 1 at each position, where is the true label of each pixel, R is a weight map about important pixels introduced in the training process.

[0108] Finally, morphological calculation is used to separate the boundary, and the weight calculation process is , is Balancing the weight of the proportion of text and noise categories, is the distance from the pixel point to the nearest text edge, is the distance from the pixel point to the second nearest text edge, and​ are constants, whose values are set to 10 and 5 respectively. Through the constraint of the loss function, the model can effectively segment the text and noise in the image.

[0109] In this embodiment, based on the existing large amount of paired data set (i.e. the stele text paired data set obtained in step 2.2), wherein represents the noisy stele image in the stele text paired data set, represents the clean stele image in the stele text paired data set corresponding to N. Figure 2 The part shown is the network structure of the model of the application, the input is input into the trained image text detection model, and through accurate frame selection and text range denoising of the stele image, the obtained stele image only contains text and noise attached around the text. The image pair is used for training the denoising network, and the result obtained by the model according to the noise image is close to the corresponding clean image . The network structure based on U-net type with skip connection is applied to guide image denoising to obtain the final denoising result.

[0110] Embodiment Two

[0111] A Chinese stele image denoising method based on text detection is executed according to the following steps:

[0112] Step 1, obtaining a stele image to be denoised;

[0113] Step 2, pre-processing the stele image to be denoised by using the method of step 2 in the Chinese stele image denoising model construction method based on text detection in embodiment 1 to obtain a processed stele paired image set and a text detection data set;

[0114] Step 3, inputting the pre-processed stele paired image set to be denoised into the stele image denoising model obtained by the Chinese stele image denoising model construction method based on text detection in embodiment 1 to obtain the corresponding clean stele image.

[0115] In this embodiment, different morphological noise stele images are used as data sets for experiments and the results are compared with other models, and two different standards are used to evaluate the generated data.

[0116] The first is to use visual effects. Most of the noise in the stele image can be perceived by the human eye, and the contrast before and after noise removal is very obvious. The second is to use evaluation indicators for objective evaluation.

[0117] The following is the calligraphy conversion result obtained by selecting different morphological noise stele images using this method and evaluating:

[0118] Dataset:

[0119] In this embodiment, ancient calligraphy and inscription images are selected as the research object, and the inscription images collected from Xi'an Stele Forest Museum are used as the research samples. Under the premise of ensuring that the inscriptions will not be damaged, a large number of classic original inscription images are shot from different angles using a high-definition camera. Due to the limitations of shooting conditions and environment, the images obtained by shooting need to be selected. When selecting, the inscription images that are not affected by external factors are selected to ensure the authenticity of the inscription images as much as possible. The denoising of the inscription images is based on the paired data set of the inscription obtained in step two, and finally the paired data sets with the number of 5524 and 700 are used for the training and testing of the inscription denoising algorithm. Figure 3 The inscription original image and the paired data of the processed inscription are shown, wherein (a) is the size-unified inscription image, the noise image and the clean image obtained in step two are (b) and (c) respectively.

[0120] Figure 4 The text detection data pair is shown, that is, the text detection image obtained in step two, (a) is the result of projecting the clean inscription image in different directions, that is, the sum of the rows and columns of the matrix obtained by converting the clean image in the horizontal direction, (b) is the result of text frame selection on the clean inscription image, and (c) is the result of frame selection in the same position on the inscription noise image. It can be found that using this method for text segmentation will frame all the Chinese characters and noise in the image.

[0121] Denoising of inscription images with speckle noise:

[0122] In this embodiment, the IDDNet method is used to predict the inscription images with speckle noise, and the result is shown in Figure 5 , wherein (a) is the inscription image to be denoised, (b) is the frame selection result obtained by the image text detection model, that is, the text frame image predicted by the noise image after the model trained by the image text detection network, (c) is the result obtained after frame-out denoising, that is, the result of denoising the range outside the image text frame after the second column, (d) is the final denoising result obtained after frame-in denoising, that is, the text image predicted and segmented by the frame-in denoising network after training, which is also the result image obtained in the last step of this experiment. It can be found that IDDNet can obviously remove the inscription noise in the image, and the visual effect of the inscription image is improved.

[0123] Denoising of inscription images with scratch noise:

[0124] In this embodiment, the IDDNet method is used to predict the inscription images with scratch noise, and the result is shown in Figure 6The results of the IDDNet experiment are compared with other image denoising methods in this embodiment, and it can be found that the method of this chapter is better than other methods in various indexes.

[0125] Denoising of a stele image with sheet corrosion noise:

[0126] In this embodiment, the IDDNet method is used to predict the stele image with sheet corrosion noise, and the results are shown in Figure 7 (a) is the stele image to be denoised, (b) is the frame selection result obtained by the image text detection model, (c) is the result obtained after frame-out denoising, and (d) is the final denoising result obtained after frame-in denoising, that is, the result image obtained in the last step of this experiment. It can be seen that for images with a certain distance between text and noise, IDDNet can also solve the problem well even if the noise is serious. After the text position is determined by frame selection, most of the noise in the image can be directly removed, which is a direct manifestation of the advantage of IDDNet.

[0127] Evaluation index:

[0128] To ensure the objectivity and fairness of the results, multiple image indexes are used for evaluation in this experiment, including peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learning perceptual image patch similarity (LPIPS), and root mean square error (RMSE). The differences between the experimental results and the clean images are compared by different calculation methods. The larger the PSNR and SSIM values, the better the image denoising effect; the smaller the LPIPS and RMSE values, the better the image denoising effect.

[0129] In this embodiment, the results of the IDDNet experiment are compared with other image denoising methods, and it can be found that the method of this chapter is better than other methods in various indexes.

[0130] Table 3 Experimental index results

[0131]

[0132] This is because other methods lose too much text detail when removing complex noise such as scratches. Although the visual effect of some images is not very obvious, it is difficult to distinguish whether the text information is missing, but using objective indexes to compare the pixels of the image pair gets a lower calculation result, and the effect needs to be improved. The IDDNet method of the present application removes most of the noise without losing text details after determining the text range of the image by text detection, so the denoising effect is obviously improved.

[0133] Embodiment three

[0134] The embodiment provides a Chinese stele text image denoising model construction device, which comprises a stele text image acquisition module, a preprocessing module and a model training module connected in sequence.

[0135] The stele text image acquisition module is used for acquiring multiple complete stele text images with noise to obtain a noise data set.

[0136] The preprocessing module is used for preprocessing the noise data set to obtain a preprocessed noise data set, artificially processing the preprocessed data set to obtain a clean data set, and performing text frame selection processing on the clean data set to obtain a corresponding text detection data set.

[0137] The model training module is used for taking the preprocessed noise images in the text detection data set obtained by the preprocessing module as input, taking the corresponding noise images after frame selection as reference output, training an image text detection model to obtain a trained image text detection model, inputting the preprocessed noise images in the text detection data set into the trained image text detection model to obtain a prediction result and performing out-of-frame denoising on the prediction result, taking the prediction result as input and taking the clean data set obtained by the preprocessing module as reference output, training an in-frame denoising model to obtain a trained in-frame denoising model, and finally obtaining a Chinese stele text image denoising model.

[0138] The preprocessing module is used for performing the following steps:

[0139] Step 2.1: all stele text images in the noise data set are uniformly scaled to 512*512 size pixels to obtain adjusted stele text images.

[0140] Step 2.2: the adjusted stele text images are binarized to obtain binarized stele text images.

[0141] Step 2.3: according to the presentation of the stele text, morphological operation is performed on the binarized stele text images to obtain repaired stele text images.

[0142] Step 2.4: artificial denoising is performed on all the repaired stele text images to obtain corresponding clean data sets, and the preprocessed noise data set obtained in step 2.2 is paired to form a stele text paired data set.

[0143] Step 2.5: frame selection processing is performed on all the stele text images in the clean data set obtained in step 2.4 to obtain noise images after frame selection, and the noise images after frame selection are paired with the preprocessed noise images obtained in step 2.2 to form a text detection data set.

[0144] The first six layers of the image text detection model comprise convolution layers, activation function layers, Batch Norm layers and pooling layers connected in sequence, and the seventh to tenth layers comprise convolution layers, Batch Norm layers, convolution layers and Batch Norm layers connected in sequence; the output of the second layer and the output of the ninth layer are connected in the element direction for jump connection transmission, the output of the third layer and the output of the eighth layer, the output of the fourth layer and the output of the seventh layer, and the output of the fifth layer and the output of the sixth layer are respectively connected in the jump connection; the convolution kernel size of the convolution layers in the first to sixth layers is all set to 3*3, the convolution kernel size of the pooling layers is all set to 2*2, and the step is also set to 2; the first convolution kernel size of the seventh to tenth layers is set to 1*1, and the second is set to 3*3.

[0145] The first four layers of the in-frame denoising model all comprise convolution layers, convolution layers and pooling layers connected in sequence, the fifth layer is a convolution layer, a convolution layer and a Batch Norm layer connected in sequence, and the sixth to ninth layers are an up-sampling layer, a convolution layer and a convolution layer connected in sequence, wherein the output of the first layer and the output of the ninth layer are connected in the element direction for jump connection transmission, the output of the second layer and the output of the eighth layer, the output of the third layer and the output of the seventh layer, and the output of the fourth layer and the output of the sixth layer are respectively connected in the jump connection; the convolution kernel size of the convolution layers in the first to fourth layers is all set to 3*3, the convolution kernel size of the pooling layers is all set to 2*2, and the step is also set to 2; the up-sampling layer of the sixth to ninth layers is realized by using the deconvolution of a 2*2 kernel, and the convolution kernel size of the convolution layer is set to 3*3.

[0146] Embodiment four

[0147] The embodiment provides a Chinese stele text image denoising device, comprising a to-be-denoised stele text image acquisition module, a to-be-denoised stele text image preprocessing module and a denoising module connected in sequence;

[0148] The to-be-denoised stele text image acquisition module is used for obtaining a to-be-denoised stele text image;

[0149] The to-be-denoised stele text image preprocessing module is used for preprocessing the to-be-denoised stele text image by using the preprocessing module in the Chinese stele text image denoising model construction device, to obtain a set of preprocessed stele text image pairs and a text detection data set;

[0150] The denoising module is used for inputting the set of preprocessed to-be-denoised stele text image pairs into the Chinese stele text denoising model obtained by the Chinese stele text image denoising model construction device for processing, to obtain a corresponding clean stele text image.

Claims

1. A method for constructing a Chinese stele text image denoising model, characterized in that, Specifically comprising the following steps: Step 1, collect multiple complete noisy stele image, obtain noise data set; Step 2, the noise data set is pretreated, and the pretreated noise data set is obtained, and the pretreated noise data set is artificially processed to obtain a clean data set; the clean data set is processed by character frame selection to obtain a corresponding character detection data set; specifically comprising the following sub steps: Step 2.1: all stele images in the noise data set are uniformly scaled to 512*512 size pixels, and the adjusted stele image is obtained; Step 2.2: the adjusted stele image is binarized to obtain a binarized stele image; Step 2.3: according to the appearance of the stele, the binarized stele image is subjected to morphological operation to obtain a repaired stele image; Step 2.4: all repaired stele images are artificially denoised to obtain a corresponding clean data set, which is paired with the pretreated noise data set obtained in step 2.2 to form a stele paired data set; Step 2.5: all stele images in the clean data set obtained in step 2.4 are subjected to frame selection processing to obtain a frame selected noise image, and the frame selected noise image is paired with the pretreated noise image obtained in step 2.2 to form a character detection data set; Step 3, the pretreated noise image in the character detection data set obtained in step 2 is taken as input, and the corresponding frame selected noise image is taken as reference output, the image character detection model is trained, and the trained image character detection model is obtained; the pretreated noise image in the character detection data set is input into the trained image character detection model, the prediction result is obtained and the out-of-frame denoising is carried out, and it is taken as input, the clean data set obtained in step 2 is taken as reference output, the in-frame denoising model is trained, and the trained in-frame denoising model is obtained, finally the Chinese stele image denoising model is obtained; When training the image text detection model, the following formula is used as a loss function : wherein, a stele noise image representing the bounding text region predicted by the image text detection model, is the bounding noise image obtained in step 2.5, represents an image pixel point; is a confidence reference matrix: wherein, representing the number of text boxes in the epitaph image predicted by the network, representing the number of text boxes obtained from the noise image after frame selection in the text detection data set obtained in step 2.5; for the segmented text box, representing the text box region, representing the image pixel point, and the pixel points belonging to the confidence score of the pixel points in the range are all set to and the rest are set to 1; In training the denoising model in the bounding box, the following formula is used as a loss function : where the softmax loss function where, denotes the activation value in the feature channel at pixel , , , is the number of classes, is the approximated max function; The loss function is then used to penalize deviation from 1, where is the true label of each pixel, R is a weight map introduced during training regarding important pixels; The weight calculation process is , The first six layers of the image character detection model respectively include convolution layer, activation function layer, BatchNorm layer and pooling layer connected in sequence, the seventh to tenth layers respectively include convolution layer, Batch Norm layer, convolution layer and BatchNorm layer connected in sequence; the output of the second layer and the output of the ninth layer are connected in the element direction and transmitted by jumping connection, the output of the third layer and the output of the eighth layer, the output of the fourth layer and the output of the seventh layer, and the output of the fifth layer and the output of the sixth layer are respectively connected by jumping connection; the convolution kernel size in the convolution layer in the first to sixth layers is all set to 3*3, the pooling kernel size in the pooling layer is all set to 2*2, and the step is also set to 2; the first convolution kernel size of the seventh to tenth layers is set to 1*1, and the second is set to 3*3; Balancing the weight of the proportion of text and noise categories, is the distance from the pixel point to the nearest text edge, is the distance from the pixel point to the second nearest text edge, and are constants, and the values are set to 10 and 5, respectively; ​ The first four layers of the in-frame denoising model each include a convolution layer, a convolution layer and a pooling layer connected in sequence, the fifth layer is a convolution layer, a convolution layer and a Batch Norm layer connected in sequence, and the sixth to ninth layers are each an up-sampling layer, a convolution layer and a convolution layer connected in sequence, wherein the output of the first layer and the output of the ninth layer are connected in a skip connection in an element direction, the output of the second layer and the output of the eighth layer, the output of the third layer and the output of the seventh layer, and the output of the fourth layer and the output of the sixth layer are respectively connected in a skip connection; the convolution kernel size of the convolution layer in the first to fourth layers is each set to 3x3, the pooling kernel size of the pooling layer is each set to 2x2, and the step is also set to 2; the up-sampling layer of the sixth to ninth layers is implemented by using a 2x2 kernel deconvolution, and the convolution kernel size of the convolution layer is set to 3x3.

2. A Chinese inscriptions image denoising method, characterized in that, The following steps are performed: Step one, obtaining a stele image to be denoised; Step two, preprocessing the stele image to be denoised using the method of step 2 in the Chinese stele image denoising model construction method of claim 1, to obtain a preprocessed stele pair image set and a text detection data set; Step three, inputting the preprocessed stele pair image set to be denoised into the Chinese stele image denoising model obtained by the Chinese stele image denoising model construction method of claim 1 for processing, to obtain a corresponding clean stele image.

3. A Chinese stele text image denoising model construction device, characterized in that, It comprises a stele image acquisition module, a preprocessing module and a model training module connected in sequence; The stele image acquisition module is used to acquire multiple complete noisy stele images to obtain a noise data set; The preprocessing module is used to preprocess the noise data set to obtain a preprocessed noise data set, and to obtain a clean data set by manually processing the preprocessed data set; the clean data set is subjected to text frame selection processing to obtain a corresponding text detection data set; The preprocessing module specifically performs the following steps: Step 2.1: uniformly scale all stele images in the noise data set to 512x512 size pixels to obtain adjusted stele images; Step 2.2: binarize the adjusted stele images to obtain binarized stele images; Step 2.3: according to the stele presentation, perform morphological operations on the binarized stele images to obtain repaired stele images; Step 2.4: manually denoise all repaired stele images to obtain a corresponding clean data set, which is paired with the preprocessed noise data set obtained in step 2.2 to form a stele pair data set; Step 2.5: frame the all stele images in the clean data set obtained in step 2.4 to obtain framed noise images, and the framed noise images are paired with the preprocessed noise images obtained in step 2.2 to form a text detection data set; The model training module is configured to take the preprocessed noise image in the text detection data set obtained by the preprocessing module as input, take the corresponding boxed noise image as reference output, train the image text detection model, and obtain a trained image text detection model; input the preprocessed noise image in the text detection data set into the trained image text detection model, obtain a prediction result, perform out-of-box denoising on the prediction result, take the prediction result as input, take the clean data set obtained by the preprocessing module as reference output, train an in-box denoising model, obtain a trained in-box denoising model, and finally obtain a Chinese stele image denoising model; When training the image text detection model, the following formula is used as a loss function : wherein, a stele noise image representing the bounding text region predicted by the image text detection model, is the bounding noise image obtained in step 2.5, represents an image pixel point; is a confidence reference matrix: wherein, representing the number of text boxes in the epitaph image predicted by the network, representing the number of text boxes obtained from the noise image after frame selection in the text detection data set obtained in step 2.5; for the segmented text box, representing the text box region, representing the image pixel point, and the pixel points belonging to the confidence score of the pixel points in the range are all set to , and the rest are set to 1; In training the denoising model within the bounding box, the formula III is used as a loss function : where the softmax loss function where, denotes the activation value in the feature channel at pixel , , , is the number of classes, is the approximated max function; The loss function is then used to penalize deviation from 1, where is the true label of each pixel, R is a weight map introduced during training about important pixels; The weight calculation process is , is Balancing the weight of the proportion of text and noise categories, is the distance from the pixel point to the nearest text edge, is the distance from the pixel point to the second nearest text edge, and are constants, the values are set to 10 and 5 respectively; The first six layers of the image text detection model respectively include a convolution layer, an activation function layer, a BatchNorm layer and a pooling layer connected in sequence, and the seventh to tenth layers respectively include a convolution layer, a Batch Norm layer, a convolution layer and a BatchNorm layer connected in sequence; the output of the second layer and the output of the ninth layer are connected in a skip connection in an element direction, the output of the third layer and the output of the eighth layer, the output of the fourth layer and the output of the seventh layer, and the output of the fifth layer and the output of the sixth layer are respectively connected in a skip connection; the convolution kernel size in the convolution layer in the first to sixth layers is all set to 3×3, the pooling kernel size in the pooling layer is all set to 2×2, and the step is also set to 2; the first convolution kernel size in the seventh to tenth layers is set to 1×1, and the second convolution kernel size is set to 3×3; The first four layers of the in-box denoising model all include a convolution layer, a convolution layer and a pooling layer connected in sequence, the fifth layer is a convolution layer, a convolution layer and a Batch Norm layer connected in sequence, and the sixth to ninth layers are all an upsampling layer, a convolution layer and a convolution layer connected in sequence, wherein the output of the first layer and the output of the ninth layer are connected in a skip connection in an element direction, the output of the second layer and the output of the eighth layer, the output of the third layer and the output of the seventh layer, and the output of the fourth layer and the output of the sixth layer are respectively connected in a skip connection; the convolution kernel size in the convolution layer in the first to fourth layers is all set to 3×3, the pooling kernel size in the pooling layer is all set to 2×2, and the step is also set to 2; the upsampling layer in the sixth to ninth layers is implemented by using a 2×2 kernel deconvolution, and the convolution kernel size of the convolution layer is set to 3×3.

4. A Chinese stele text image denoising device, characterized in that, The Chinese stele image denoising model construction device includes a preprocessing module, an image text detection model training module, an in-box denoising model training module and a Chinese stele image denoising model; The preprocessing module is configured to preprocess a Chinese stele image to obtain a preprocessed Chinese stele image; The preprocessing module is configured to preprocess a Chinese stele image to obtain a preprocessed Chinese stele image; The denoising model is configured to input the preprocessed Chinese stele image into the Chinese stele denoising model obtained by the Chinese stele image denoising model construction device to obtain a corresponding clean Chinese stele image.

Images