A method, device and computer equipment for removing specific shape noise in an image

Abstract

The application relates to a specific shape noise removing method in an image, comprising the following steps: generating an image data set containing specific shape noise according to the characteristics of the specific shape noise in the image; training and saving an image segmentation network for specific shape noise segmentation of the image; performing segmentation processing on the image in the image data set by using the image segmentation network; training and saving a generative adversarial network for removing the specific shape noise; inputting the image segmented by using the image segmentation network into the generative adversarial network; and removing the specific shape noise in the image by using the generative adversarial network. The application can effectively remove the specific shape noise in the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image detection and recognition, and particularly relates to a specific shape noise removing method and device in an image and a computer device. BACKGROUND

[0002] The rapid development of the computer vision field requires a large amount of image data as support. In a scanning scene, an image usually has good readability. However, the source of the image usually has diversity, such as an image captured from a natural scene or an image scanned. In a natural capturing scene, the image is usually captured by a mobile phone or the like. These images often contain distortion, dirt and the like. Due to the fact that some specific shape noise is contained in part of the image, unnecessary interference is caused to the feature description of the image. In addition, the specific shape noise is mainly composed of shape pixel points with a closed curve feature in the image, and the specific shape noise also has some interference to the features in the image.

[0003] Therefore, how to provide a specific shape noise removing method in an image is a problem to be solved at present. SUMMARY

[0004] The embodiment of the present application provides a specific shape noise removing method in an image, so as to solve the problem that the specific shape noise in the image causes interference to the features in the image in the prior art. In order to have a basic understanding of some aspects of the disclosed embodiments, the following is a brief summary. The summary part is not a general review, nor is it intended to determine the key / important constituent elements or delineate the protection scope of these embodiments. The only purpose is to present some concepts in a simple form as a prelude to the detailed description below.

[0005] In a first aspect, the present application provides a specific shape noise removing method in an image, comprising the following steps:

[0006] generating an image data set containing specific shape noise according to the characteristics of the specific shape noise in the image;

[0007] training and saving an image segmentation network for specific shape noise segmentation of the image, and performing segmentation processing on the image in the image data set by using the image segmentation network;

[0008] training and saving a generative adversarial network for removing specific shape noise, inputting the image segmented by using the image segmentation network into the generative adversarial network, and removing the specific shape noise in the image by using the generative adversarial network.

[0009] Optionally, the step of generating an image data set containing specific shape noise according to the characteristics of the specific shape noise in the image comprises the following steps:

[0010] Randomly select a specific shape, control the size of the figure, and batch generate a binary mask image dataset containing specific shape noise;

[0011] Randomly select several pictures as background pictures, and fuse the images in the binary mask image dataset containing specific shape noise into the background pictures respectively to form a synthetic noise image dataset;

[0012] Collect images containing specific shape noise in an actual scene, label the specific shape noise positions in the images, and construct a labeled image dataset.

[0013] Optionally, the generating step of the binary mask image dataset containing specific shape noise comprises: randomly selecting a shape, making the size of the figure complete and center-aligned in a 512*512 pixel background, and randomly translating and / or rotating the specific shape noise.

[0014] Optionally, the generating step of the synthetic noise image dataset comprises:

[0015] Randomly select a background image, and fuse the binary mask image containing specific shape noise and the background image by using the corresponding pixel value weighted addition method;

[0016] Respectively read the three-channel image data of the fused image, perform uniform sampling, and use the sampling value as the pixel weight value of the binary mask image;

[0017] Subtract 1 from the pixel weight value to obtain the pixel weight value of the background image, and then perform weighted summation on the pixels of the binary mask image and the background image to obtain a synthetic noise image.

[0018] Optionally, the step of respectively reading the three-channel image data of the fused image and performing uniform sampling, wherein the sampling range is 0.2 to 0.8.

[0019] Optionally, the step of constructing a labeled image dataset comprises: collecting images containing specific shape noise in an actual scene, circling the edges of the specific shape noise, filling the specific shape noise region pixels with 0, and filling the non-noise region pixels with 255.

[0020] Optionally, the step of training and saving the image segmentation network for specific shape noise segmentation of images comprises the following steps:

[0021] Randomly select M binary mask images of a specific shape, and randomly select M background images, respectively, and fuse them with the M binary mask images generated one by one to generate M synthetic noise images, and take a binary mask image and the synthetic noise image generated therefrom as a group of input data, a total of M groups of data; collect N images in the actual scene, and respectively label the noise positions thereof to construct N labeled images, and take an image collected in the actual scene and the labeled image constructed therefrom as a group of input data, a total of N groups of data, and M and N are positive integers.

[0022] Optionally, the step of training and saving the image segmentation network for specific shape noise segmentation of the image further comprises the following steps: preprocessing the synthetic noise image, reading the BGR three-channel image thereof, performing size processing on the read image, uniformly normalizing the size of the image to 512*512 pixels, and performing a data enhancement operation with a probability of 0.2-0.4 on the normalized image, performing a horizontal flip and a rotation operation with a rotation angle not exceeding 10 degrees on the data enhanced image, and then converting the rotated image into an RGB channel image.

[0023] Optionally, the step of training and saving the image segmentation network for specific shape noise segmentation of the image comprises the following steps:

[0024] Constructing an image segmentation network for specific shape noise segmentation of the image, and defining an image segmentation loss function;

[0025] Segmenting the synthetic noise image using the image segmentation network to output a binary mask image after segmenting the specific shape noise;

[0026] Optimizing the image segmentation network using a stochastic gradient descent optimizer, and training the image segmentation network using the image segmentation loss function;

[0027] After J times of segmentation training iterations of the image segmentation network on the synthetic noise image data, saving the first network weight, initializing with the first network weight, training the initialized image segmentation network on the labeled image, saving the second network weight after K times of iterations, and J and K are positive integers.

[0028] Optionally, the step of training and saving the image segmentation network for specific shape noise segmentation of the image further comprises:

[0029] Using the second network weight to test new sample images that do not participate in training.

[0030] Optionally, the step of training and saving the generative adversarial network for removing specific shape noise comprises the following steps:

[0031] The specific shape noise removal generative adversarial network is constructed, and a generative adversarial loss function is defined.

[0032] The specific shape noise removal generative adversarial network is constructed, and a generative adversarial loss function is defined.

[0033] The binary mask image obtained by using the first network weight of the image segmentation network is input into the generative adversarial network for network iteration P times, and the third network weight is saved after the network iteration, the third network weight is used for initialization, the initialized generative adversarial network is used for training of the labeled image, the binary mask image obtained by using the second network weight of the image segmentation network is input into the generative adversarial network for network iteration Q times, and the fourth network weight is obtained, and P and Q are positive integers.

[0034] Optionally, the generative adversarial network comprises a generator and a discriminator, and the generative adversarial loss function comprises a generator loss function and a discriminator loss function.

[0035] Optionally, the step of training and saving the generative adversarial network for removing specific shape noise further comprises: using the fourth network weight to test new sample images that do not participate in training.

[0036] Optionally, when the image segmentation network and the generator are trained, the learning rate is set to 2e-5 to 4e-5, and when the discriminator is trained, the learning rate is set to 5e-6 to 2e-5.

[0037] In a second aspect, the present application provides an image specific shape noise removal device, which adopts the steps of the image specific shape noise removal method according to any one of the above aspects to remove specific shape noise in the image.

[0038] In a third aspect, the present application provides a computer device, which comprises a memory, a processor and a program stored in the memory and executable on the processor, and the processor implements the steps of the image specific shape noise removal method according to any one of the above aspects when executing the program.

[0039] The technical scheme provided by the embodiments of the present application can have the following beneficial effects:

[0040] The specific shape noise image dataset is generated for network training, and the image segmentation network is used for segmentation processing, and then the generative adversarial network is trained by using the image segmented by the image segmentation network, so that the network can accurately learn the position of the specific shape noise, has better denoising capability and stronger robustness.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] Figure 1 This is a flowchart illustrating a method for removing noise of a specific shape in an image according to an exemplary embodiment;

[0044] Figure 2 This is a schematic diagram of the structure of an image segmentation network according to an exemplary embodiment;

[0045] Figure 3 This is a schematic diagram of the structure of a generative adversarial network according to an exemplary embodiment;

[0046] Figure 4 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0047] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0048] The terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing this paper and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. In the description herein, unless otherwise specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be directly connected, or indirectly connected through an intermediate medium, and the specific meaning of the above terms can be understood by the person skilled in the art according to the specific circumstances.

[0049] In this paper, unless otherwise specified, the term "a plurality of" means two or more.

[0050] In this paper, the character " / " represents that the front and rear objects are in an "or" relationship. For example, A / B represents: A or B.

[0051] In this paper, the term "and / or" is a description of the relationship between the objects, which means that there can be three relationships. For example, A and / or B, which means: A or B, or, A and B, three relationships.

[0052] The embodiments in this application and the features in the embodiments can be combined with each other without conflict.

[0053] As Figure 1 As shown in the figure, the image specific shape noise removal method provided by the embodiments of the application includes the following steps: generating an image data set containing specific shape noise according to the characteristics of the specific shape noise in the image; training and saving an image segmentation network for specific shape noise segmentation of the image, and using the image segmentation network to segment the image in the image data set; training and saving a generative adversarial network for removing specific shape noise, inputting the image segmented by the image segmentation network into the generative adversarial network, and using the generative adversarial network to remove the specific shape noise in the image.

[0054] In one embodiment, the step of generating an image data set containing specific shape noise according to the characteristics of the specific shape noise in the image includes:

[0055] Randomly select a shape, and reasonably control the size of the figure so that it can be placed in the center of the 512*512 pixel background in its entirety, then randomly translate and rotate the specific shape noise.

[0056] Randomly select a background image, and fuse the binary mask image containing the specific shape noise with the background image by using the corresponding pixel value weighted addition method. Read the image data of three channels respectively, then uniformly sample from 0.2 to 0.8, and use the sampling value as the pixel weight value of the binary mask image. Subtract the weight value from 1 to obtain the pixel weight value of the background image. Finally, perform weighted summation on the pixels of the two images to obtain the final synthesized noise image.

[0057] Optionally, a background image is randomly selected, and the binary mask image containing the specific shape noise is fused with the background image by using the corresponding pixel value weighted addition method. The image data of three channels is read respectively, then uniformly sampled from 0.3 to 0.7, and the sampling value is used as the pixel weight value of the binary mask image. Subtract the weight value from 1 to obtain the pixel weight value of the background image. Finally, perform weighted summation on the pixels of the two images to obtain the final synthesized noise image.

[0058] An image in an actual scene is collected, the image is manually labeled, the edge of the specific shape noise is circled, and the pixel value of the specific shape noise region is filled with 0 and the pixel value of the non-noise region is filled with 255.

[0059] In one embodiment, the step of generating an image data set containing specific shape noise according to the characteristics of the specific shape noise in the image comprises:

[0060] According to the specific shape, a binary mask image containing specific shape noise is generated, and the size of the graph is reasonably controlled. A background image is selected, padded to 512*512 pixels, and the binary mask image containing specific shape noise is fused onto the background image. A large number of binary mask images of specific shape noise are batch-generated, a background image is randomly selected and fused into the background image while retaining the annotation information, and a specific shape noise synthetic image data set is constructed. Image data containing specific shape noise in an actual scene is collected, and Photoshop or other similar software is used for manual annotation of the specific shape noise position to construct a labeled image data set.

[0061] In one embodiment, the step of training and saving an image segmentation network for specific shape noise segmentation of an image comprises:

[0062] Data preparation: randomly select 1000-3000 binary mask images of a specific shape, randomly select 1000-3000 background images for synthesis operation, generate synthetic noise images, take one binary mask image and the synthetic noise image generated therefrom as a group of input data, a total of 1000-3000 groups of data; collect 50-100 images in the actual scene, label the specific shape noise using Photoshop or other similar software, take the labeled image and the original image collected in the actual scene as a group of input data, a total of 50-100 groups of data;

[0063] Optionally, randomly select 2000 binary mask images of a specific shape, randomly select 2000 background images for synthesis operation, generate 2000 synthetic noise images, take one generated binary mask image and the synthetic noise image generated therefrom as a group of input data, a total of 2000 groups of data; collect 70 images in the actual scene, manually label the specific shape noise using Photoshop or other similar software, take one original image collected in the actual scene and the labeled image constructed therefrom as a group of input data, a total of 70 groups of data.

[0064] Data processing: pre-process the synthetic noise image, first read the BGR three-channel image, process the image size, normalize it to 512*512 size, and perform data enhancement operation on the image with a probability of 0.2-0.4, perform horizontal flip and rotation operation with a rotation angle not exceeding 10 degrees, finally, convert the rotated image to an RGB channel image.

[0065] Optionally, the probability of the above image data enhancement operation is 0.3.

[0066] Construct an image segmentation network SegUnet for specific shape noise segmentation of the image, please refer to Figure 2The image segmentation network SegUnet in the embodiment has a U-shaped network structure, and an image after data processing is input into the segmentation network. An a1 is obtained through convolution (64 3*3 convolution kernels, and a step length of 2) and Leaky ReLU (Leaky Rectified Linear Unit, which is a kind of activation function, solves the problem of parameter updating in the network back propagation process, accelerates the convergence of the model, and solves the problem of gradient disappearance) operation. An a2 is obtained through convolution (128 3*3 convolution kernels, and a step length of 2) and Leaky ReLU operation on the a1. An a3 is obtained through convolution (256 3*3 convolution kernels, and a step length of 2) and Leaky ReLU operation on the a2. An a4 is obtained through convolution (512 3*3 convolution kernels, and a step length of 2) and Leaky ReLU operation on the a3. A d1 is obtained through dilated convolution (512 3*3 convolution kernels, a step length of 1, and an expansion ratio of 2) and Leaky ReLU operation on the a4. A d2 is obtained through dilated convolution (512 3*3 convolution kernels, a step length of 1, and an expansion ratio of 4) and Leaky ReLU operation on the d1. A d3 is obtained through dilated convolution (512 3*3 convolution kernels, a step length of 1, and an expansion ratio of 8) and Leaky ReLU operation on the d2. A d4 is obtained through dilated convolution (512 3*3 convolution kernels, a step length of 1, and an expansion ratio of 16) and Leaky ReLU operation on the d3. A b1 is obtained through deconvolution (256 3*3 convolution kernels, a step length of 1), Leaky ReLU operation, and connection with the a4 layer. A b2 is obtained through deconvolution (128 3*3 convolution kernels, a step length of 1), Leaky ReLU operation, and connection with the a3 layer. A b3 is obtained through deconvolution (64 3*3 convolution kernels, a step length of 1), Leaky ReLU operation, and connection with the a2 layer. A b4 is obtained through deconvolution (1 3*3 convolution kernel, a step length of 1), Sigmoid operation, and connection with the a1 layer. Finally, a binary mask image after segmentation of specific shape noise is output.

[0067] A loss function is defined, and the calculation formula is as follows:

[0068]

[0069] In the above formula, the first term is x,y G x,y log(P x,y ) is a cross-entropy loss, and the second term is a Dice loss, where P x,y and G x,yrespectively represent the pixel value of the image segmentation network output image and the true value at the image (x, y), and λ B and λ D are hyperparameters, respectively representing the weight value of its loss;

[0070] The above image segmentation network SegUnet is optimized by using the Adam stochastic gradient descent optimizer, and is trained by using the above loss function. After the image segmentation network SegUnet is iterated 50-200 times on the synthetic noise image data, the first network weight is saved. Then, the first network weight is used for initialization, the manually labeled image data is trained, and the image segmentation network SegUnet is used for training. After the image segmentation network SegUnet is iterated 500-2000 times, the second network weight is saved.

[0071] Optionally, after the image segmentation network SegUnet is iterated 100 times on the synthetic noise image data, the first network weight is saved. Then, the first network weight is used for initialization, the manually labeled image data is trained, and the image segmentation network SegUnet is used for training. After the image segmentation network SegUnet is iterated 1000 times, the second network weight is saved.

[0072] Optionally, in an embodiment, the saved second network weight is used to test new sample images that do not participate in training.

[0073] In an embodiment, the step of training and saving the generative adversarial network for removing specific shape noise includes the following steps:

[0074] The second network weight of the saved image segmentation network SegUnet is used to process the image processed in the above embodiment, to obtain a segmented binary mask image. The binary mask image and the original image are input into the generator of the generative adversarial network as a group of data, to obtain an image after removing specific shape noise, and then the image is input into the discriminator.

[0075] In an embodiment, the generator is composed of a rough sub-network and a refining sub-network, and the structure is as shown in Figure 3As shown, the binary mask image and the original image obtained by SegUnet segmentation are input into the generative adversarial network, wherein a1 is obtained by a gating convolution (128 5*5 convolution kernels, step length 1), a Leaky ReLU operation; a2 is obtained by a gating convolution (128 5*5 convolution kernels, step length 1), a Leaky ReLU operation; a3 is obtained by a gating convolution (256 3*3 convolution kernels, step length 2), a Leaky ReLU operation; a4 is obtained by a gating convolution (256 3*3 convolution kernels, step length 1), a Leaky ReLU operation; a5 is obtained by a gating convolution (256 3*3 convolution kernels, step length 2), a Leaky ReLU operation; a6 is obtained by a gating convolution (256 3*3 convolution kernels, step length 1), a Leaky ReLU operation; a7 is obtained by a gating convolution (512 3*3 convolution kernels, step length 1), a Leaky ReLU operation; a8 is obtained by a gating convolution (512 3*3 convolution kernels, step length 2), a Leaky ReLU operation; a9 is obtained by a gating convolution (512 3*3 convolution kernels, step length 1), a Leaky ReLU operation; a10 is obtained by a gating convolution (1024 3*3 convolution kernels, step length 2), a Leaky ReLU operation; a11 is obtained by a gating convolution (1024 3*3 convolution kernels, step length 2), a Leaky ReLU operation; a12 is obtained by a gating convolution (1024 3*3 convolution kernels, step length 1), a Leaky ReLU operation; c1 is obtained by a gating deconvolution (1024 3*3 convolution kernels, step length 1), a Leaky ReLU operation, and connection with the a10 layer; c2 is obtained by a gating deconvolution (512 3*3 convolution kernels, step length 1), a Leaky ReLU operation, and connection with the a8 layer; c3 is obtained by a gating deconvolution (512 3*3 convolution kernels, step length 1), a Leaky ReLU operation, and connection with the a6 layer; c4 is obtained by a gating deconvolution (256 3*3 convolution kernels, step length 1), a Leaky ReLU operation, and connection with the a4 layer; c5 is obtained by a gating deconvolution (3 3*3 convolution kernels, step length 1); at this point, the crude network of the generator is completed; r1 is obtained by a gating convolution (128 5*5 convolution kernels, step length 1), a Leaky ReLU operation; r2 is obtained by a gating convolution (256 3*3 convolution kernels, step length 2), a Leaky ReLU operation; r3 is obtained by a gating convolution (256 3*3 convolution kernels, step length 1), a Leaky ReLU operation; r4 is obtained by a gating convolution (512 3*3 convolution kernels, step length 2), a Leaky ReLU operation;r4 is subjected to a gated convolution (512 3*3 convolution kernels, step size 1), a Leaky ReLU operation to obtain r5; r5 is subjected to a gated convolution (512 3*3 convolution kernels, step size 1), a Leaky ReLU operation to obtain r6; and then subjected to 4 times of a dilated convolution (512 3*3 convolution kernels, step size 1), and the dilated ratios are 2, 4, 8 and 16 to obtain d1, d2, d3 and d4; d4 is subjected to a gated convolution (512 3*3 convolution kernels, step size 1), a Leaky ReLU operation to obtain b1; b1 is subjected to a gated convolution (512 3*3 convolution kernels, step size 1), a Leaky ReLU operation to obtain b2; b2 is subjected to a gated convolution (512 3*3 convolution kernels, step size 1), a Leaky ReLU operation, and connected with r6 to obtain b3; b3 is subjected to a gated convolution (256 3*3 convolution kernels, step size 1), a Leaky ReLU operation, and connected with a4 to obtain b4; b4 is subjected to a gated convolution (256 3*3 convolution kernels, step size 1), a Leaky ReLU operation, and connected with r3 to obtain b5; b5 is subjected to a gated convolution (128 3*3 convolution kernels, step size 1), a Leaky ReLU operation, and connected with a2 to obtain b6; b6 is subjected to a gated convolution (128 3*3 convolution kernels, step size 1), a Leaky ReLU operation to obtain b7; b7 is subjected to a gated convolution (128 3*3 convolution kernels, step size 1) to obtain b8; finally, an image after removing the specific shape noise is output.

[0076] In one embodiment, the discriminator is a binary classification convolutional neural network, which inputs the image after removing the specific shape noise output by the generator and the real image into the discriminator, and the structure of the discriminator is a convolutional layer (64*5*5, step size 1)→spectral_norm (spectral normalization, the spectral norm of the weight matrix calculated by the power iteration method is used to scale the weight tensor, so that the training of the discriminator of the generative adversarial network is stable)→LeakyReLU→a convolutional layer (128*3*3, step size 2)→spectral_norm→LeakyReLU→a convolutional layer (256*3*3, step size 2)→spectral_norm→LeakyReLU→a convolutional layer (512*3*3, step size 2)→spectral_norm→LeakyReLU→a convolutional layer (512*3*3, step size 2)→spectral_norm→LeakyReLU→a convolutional layer (512*3*3, step size 2)→spectral_norm→LeakyReLU→a convolutional layer (1*4*4, step size 1)→Sigmoid.

[0077] The generator loss function and the discriminator loss function are defined, and the generator loss function is as follows:

[0078] Let the original image containing specific shape noise be I, the binary mask image be M, and the label image without noise be I g , the image segmentation network output image be O M , the generator rough output image be O c , and the fine output image be O R G is the generator, D is the discriminator, and S is the Sobel operator, that is, two 3*3 convolution kernels

[0079] The calculation formula of the generator loss function is represented as:

[0080] L total =λ c L c +λ G L G +λ e L e (2)

[0081] Wherein, λ c , λ G and λ e are adjustable hyperparameters

[0082] L c is the image content loss, and the calculation formula is as follows:

[0083]

[0084] Wherein, λ b is the image background weight, and λ s is the image noise weight

[0085] L G is the adversarial loss, and the calculation formula is as follows:

[0086]

[0087] Wherein, represents the expectation of I satisfying the distribution P I (I)

[0088] L e is the edge loss, and the calculation formula is as follows:

[0089] L e =||S(O R )-S(I g )||1 (5)

[0090] The calculation formula of the discriminator loss function is represented as:

[0091]

[0092] wherein, representing that the distribution is p data (I g ) of I g expectation.

[0093] The generative adversarial network is optimized by using a random gradient descent optimizer, the generative adversarial network is trained by using a generator loss function and a discriminator loss function, the binary mask image obtained by using the first network weight of the image segmentation network SegUnet is sent into the generative adversarial network, the third network weight is saved after iteration for 50-200 times, the third network weight is used for initialization, the labeled image is trained, the binary mask image obtained by using the second network weight of the image segmentation network SegUnet is sent into the generative adversarial network for training, and the fourth network weight is obtained after iteration for 500-2000 times.

[0094] In one embodiment, the binary mask image obtained by using the first network weight of the image segmentation network SegUnet is sent into the generative adversarial network, the third network weight is saved after iteration for 100 times, the third network weight is used for initialization, the labeled image is trained, the binary mask image obtained by using the second network weight of the image segmentation network SegUnet is sent into the generative adversarial network, and the fourth network weight is obtained after iteration for 1000 times.

[0095] In one embodiment, after obtaining the fourth network weight, the new sample image not participating in the training is tested by using the fourth network weight.

[0096] In one embodiment, when training the image segmentation network SegUnet and the generator, the learning rate is set to 2e-5-4e to 5, and when training the discriminator, the learning rate is set to 5e-6 to 2e-5.

[0097] Alternatively, when training the image segmentation network SegUnet and the generator, the learning rate is set to 3e-5, and when training the discriminator, the learning rate is set to 1e-5.

[0098] In one embodiment, after obtaining the fourth network weight, the image after removing the specific shape noise is obtained by using the generator, the corresponding loss value is calculated, the network parameters are updated by using back propagation until the network converges, and then the network model of the trained generative adversarial network is saved.

[0099] The method generates an image data set containing specific shape noise for training of the network, pre-processes the image data, first inputs the pre-processed image data into an image segmentation network to obtain a binary mask image for segmenting the position of the specific shape noise. Then, the original image and the segmented binary mask image are input into a generator to obtain an image after removing the specific shape noise, and are input into a discriminator. The method can more accurately enable the network to learn the position information of the specific shape noise by adding an image segmentation network before a generative adversarial network, and has strong robustness and better denoising effect.

[0100] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 4 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store static information and dynamic information data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the steps in the above method embodiments.

[0101] Those skilled in the art can understand that Figure 4 The structure shown in the above

[0102] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0103] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0105] It should be noted that the above description is only some embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure range involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, and also covers other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

[0106] In addition, although each operation is described in a specific order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments.

[0107] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for removing noise of a specific shape from an image, characterized in that, Includes the following steps: Generate an image dataset containing noise of a specific shape based on the characteristics of noise of that specific shape in the image; Train and save an image segmentation network that performs specific shape noise segmentation on images, and use the image segmentation network to segment images in the image dataset; A generative adversarial network (GAN) for removing noise of a specific shape is trained and saved. An image segmented by an image segmentation network is input into the GAN, and the GAN is used to remove noise of a specific shape from the image. The step of generating an image dataset containing noise of a specific shape based on the characteristics of noise of a specific shape in the image includes the following steps: Randomly select specific shapes, control the size of the graphics, and generate a batch of binarized mask image datasets containing noise of specific shapes; Several images are randomly selected as background images. Images from the set of binarized mask images containing noise of specific shapes are fused into the background images to synthesize a noisy image dataset. Collect images containing noise of specific shapes in real-world scenes, annotate the locations of the noise of specific shapes in the images, and construct an annotated image dataset; The steps for generating the synthetic noise image dataset include: A background image is randomly selected, and the binarized mask image containing noise of a specific shape is fused with the background image by weighted summation of corresponding pixel values; Read the three channels of the fused image data separately, perform uniform sampling, and use the sampled values ​​as pixel weights of the binarized mask image; The pixel weight is subtracted from 1 to obtain the pixel weight of the background image. Then, the pixels of the binarized mask image and the background image are weighted and summed to obtain the synthesized noise image.

2. The method for removing noise of a specific shape in an image according to claim 1, characterized in that, The steps for generating the binarized mask image dataset containing specific shape noise include: randomly selecting a shape such that the size of the graphic can be completely centered and placed on a 512*512 pixel background, and randomly translating and / or rotating the specific shape noise.

3. The method for removing noise of a specific shape in an image according to claim 1, characterized in that, The step of reading the three-channel image data of the fused image separately and performing uniform sampling, wherein the sampling range is 0.2 to 0.

8.

4. The method for removing noise of a specific shape in an image according to claim 1, characterized in that, The steps for constructing the labeled image dataset include: acquiring images containing noise of a specific shape in a real scene, drawing the edges of the noise of the specific shape, filling the pixels in the noise region of the specific shape with 0, and filling the pixel values ​​in the non-noise region with 255.

5. The method for removing noise of a specific shape in an image according to claim 1, characterized in that, The steps of training and saving the image segmentation network for segmenting image noise of a specific shape include the following steps: M binary mask images are generated by randomly selecting specific shapes. M background images are also randomly selected and merged with the generated M binary mask images one by one to generate M synthetic noise images. One binary mask image and the synthetic noise image generated by it are used as a set of input data, for a total of M sets of data. N images from the actual scene are collected and their noise locations are labeled to construct N labeled images. One collected image from the actual scene and the labeled image constructed by it are used as a set of input data, for a total of N sets of data, where M and N are both positive integers.

6. The method for removing noise of a specific shape in an image according to claim 5, characterized in that, The steps of training and saving the image segmentation network for specific shape noise segmentation of images further include the following steps: preprocessing the synthetic noise image, reading its BGR three-channel image, resizing the read image to normalize it to a size of 512*512 pixels, performing data augmentation operation with a probability of 0.2-0.4 on the normalized image, performing horizontal flipping and rotation operation with a rotation angle not exceeding 10 degrees on the data augmented image, and then converting the rotated image into an RGB channel image.

7. The method for removing noise of a specific shape in an image according to claim 1, characterized in that, The step of training and saving the image segmentation network for segmenting image noise of a specific shape includes the following steps: Construct an image segmentation network that performs specific shape noise segmentation on an image, and define the image segmentation loss function; The image segmentation network described above is used to segment the synthetic noise image and output a binarized mask image after segmenting noise of a specific shape. The image segmentation network is optimized using a stochastic gradient descent optimizer, and the image segmentation network is trained using an image segmentation loss function. On the synthetic noisy image data, the image segmentation network is used for segmentation training J times and its first network weight is saved. The first network weight is used for initialization, and the initialized image segmentation network is used to train the labeled image segmentation. After K iterations, its second network weight is saved. J and K are both positive integers.

8. The method for removing noise of a specific shape in an image according to claim 7, characterized in that, The step of training and saving the image segmentation network for performing shape-specific noise segmentation on images further includes: The new sample images that were not used in training are tested using the second network weights.

9. The method for removing noise of a specific shape in an image according to claim 8, characterized in that, The step of training and saving the generative adversarial network with specific shape noise removed includes the following steps: Construct a generative adversarial network for noise removal of specific shapes and define a generative adversarial loss function; The generative adversarial network is optimized using stochastic gradient descent and trained using a generative adversarial loss function. For synthesized noisy image data, the binarized mask image obtained by using the first network weight of the image segmentation network is input into the generative adversarial network for P iterations, and its third network weight is saved. The third network weight is used for initialization, and the initialized generative adversarial network is used to train the labeled image. The binarized mask image obtained by using the second network weight of the image segmentation network is input into the generative adversarial network for Q iterations, and the fourth network weight is obtained. P and Q are both positive integers.

10. The method for removing noise of a specific shape in an image according to claim 9, characterized in that, The generative adversarial network includes a generator and a discriminator, and the generative adversarial loss function includes a generator loss function and a discriminator loss function.

11. The method for removing noise of a specific shape in an image according to claim 10, characterized in that, The step of training and saving the generative adversarial network with specific shape noise removed also includes: testing new sample images that were not involved in the training using the fourth network weights.

12. The method for removing noise of a specific shape in an image according to claim 10, characterized in that, When training the image segmentation network and the generator, the learning rate is set to 2e-5 to 4e-5, and when training the discriminator, the learning rate is set to 5e-6 to 2e-5.

13. A device for removing noise of a specific shape in an image, characterized in that, The device employs the steps of the image-specific shape noise removal method as described in any one of claims 1 to 12 to remove image-specific shape noise.

14. A computer device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the image-specific shape noise removal method as described in any one of claims 1 to 12.

Images