A moiré detection method and device

Abstract

The present invention discloses a moiré detection method, comprising the following steps. Step S1: preparing a training data set for a neural network; the training data set includes four categories - scaled images, cropped images, channel-stitched images, and cropped-stitched images; all images are manually annotated to determine whether they have moiré. Step S2: using the four categories in the training data set to respectively train four deep neural networks for classification to obtain four moiré detection modules. Step S3: for an input image to be judged whether it has moiré, first obtain a scaled image, a cropped image, a channel-stitched image, and a cropped-stitched image based on the input image, and then send them to four moiré detection modules to obtain four probability values, take an average of the four probability values, and obtain the final moiré determination result based on the average. The moiré detection result obtained by the present invention is more accurate.

Description

Technical Field

[0001] The invention relates to a method for detecting whether moiré patterns exist in a picture. Background Art

[0002] In digital photography, moiré patterns (also known as moiré fringe, moiré fringes, or screen patterns) refer to irregular streaks that appear in the captured image when the spatial distribution frequency of the pixels in the photographic device's photosensitive component is close to the spatial distribution frequency of the fringes in the scene being photographed. For example, when using a mobile phone camera to capture a computer display, irregular streaks are often observed in the captured image. Moiré patterns can interfere with the original color, shape, and clarity of the screen area in the image, hindering the user's ability to read and appreciate the image.

[0003] It is very important to detect whether there is moiré in the image, and after identification, the moiré can be removed later. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to more accurately identify whether there is moiré in an image.

[0005] To address the above-mentioned technical problems, the present invention proposes a moiré detection method comprising the following steps. Step S1: Prepare a training dataset for a neural network; the training dataset includes four categories: scaled images, cropped images, channel-stitched images, and cropped-stitched images; all images are manually annotated to determine whether they contain moiré. First, capture or acquire an original image; a portion of the original image contains moiré, while the remaining original image does not. Resize the original image to a size a×b, obtaining a scaled image. Crop the original image, retaining only the a×b size of the central region of the original image, to obtain a cropped image. Superimpose the a×b×3 color space channels of the scaled image and the a×b×3 color space channels of the cropped image to form an a×b×6 color space channel, referred to as a channel-stitched image. Halve the scaled image horizontally and the cropped image horizontally, and then stitch the two together horizontally to obtain a cropped-stitched image; or halve the scaled image vertically and the cropped image vertically, and then stitch the two together vertically to obtain a cropped-stitched image. The values ​​of a and b both range from 200 pixels to 500 pixels. Step S2: Using the four categories in the training data set—scaled image, cropped image, channel-stitched image, and cropped-stitched image—four deep neural networks for classification are trained to obtain four moiré detection modules: a scaled image moiré detection module, a cropped image moiré detection module, a channel-stitched image moiré detection module, and a cropped-stitched image moiré detection module. Step S3: For an input image to be determined whether it has moiré, a scaled image, a cropped image, a channel-stitched image, and a cropped-stitched image are first obtained based on the input image. These images are then fed into the scaled image moiré detection module, the cropped image moiré detection module, the channel-stitched image moiré detection module, and the cropped-stitched image moiré detection module to obtain four probability values. The four probability values ​​are averaged, and a final moiré determination result is obtained based on the average.

[0006] Preferably, a=b.

[0007] Furthermore, the original image is manually annotated to determine whether it has moiré patterns; the zoomed image, cropped image, channel-stitched image, cropped and spliced ​​image, or the moiré pattern annotation of the original image is inherited, or the moiré pattern is manually reviewed again and updated.

[0008] Preferably, if the width of the original image is less than a, the original images are first spliced ​​horizontally until the width of the spliced ​​image exceeds a, and then the spliced ​​image is cropped to retain only the a×b size of the center area to obtain a cropped image; if the height of the original image is less than b, the original images are first spliced ​​vertically until the height of the spliced ​​image exceeds b, and then the spliced ​​image is cropped to retain only the a×b size of the center area to obtain a cropped image.

[0009] Preferably, in step S2, the neural network adopts a residual network ResNet or a MobileNet neural network.

[0010] Furthermore, in step S2, the training method of the neural network is: input an image of a certain category in the training data set, and output a probability value between 0 and 1, which is used to indicate one of the presence of moiré in the image and the absence of moiré in the image, so that the output of the neural network is consistent with the manual annotation of the moiré in the image as much as possible.

[0011] Preferably, in step S2, when the neural network is trained, the parameters of the neural network are optimized using stochastic gradient descent SGD or adaptive moment estimation Adam algorithm.

[0012] Furthermore, in step S2, the last layer of the neural network is replaced by an n-class fully connected layer with a two-class fully connected layer. The two-class fully connected layer outputs a two-dimensional vector, the value of which represents the selection of one of the only two classes. The two-dimensional vector is also normalized to a probability value between 0 and 1.

[0013] Preferably, in step S3, if the average value is greater than 0.5, it indicates that there are moiré patterns in the image; if the average value is less than or equal to 0.5, it indicates that there are no moiré patterns in the image.

[0014] The present invention also provides a moiré detection device, comprising a training data generation unit, a neural network training unit, and a moiré detection unit. The training data generation unit is used to generate a training dataset for the neural network; the training dataset includes four categories: scaled images, cropped images, channel-stitched images, and cropped-stitched images. First, an original image is captured or acquired, wherein a portion of the original image exhibits moiré and the remaining original image does not. The original image is resized to a size a×b to obtain a scaled image. The original image is cropped, retaining only the a×b size of the central region of the original image to obtain a cropped image. The a×b×3 color space channels of the scaled image and the a×b×3 color space channels of the cropped image are superimposed to form a a×b×6 color space channel, referred to as a channel-stitched image. The scaled image is halved horizontally and the cropped image is halved horizontally, and the two are stitched together horizontally to obtain a cropped-stitched image; alternatively, the scaled image is halved vertically and the cropped image is halved vertically, and the two are stitched together vertically to obtain a cropped-stitched image. The values ​​of a and b both range from 200 pixels to 500 pixels. The neural network training unit is used to train four deep neural networks for classification using the four categories in the training data set - scaled image, cropped image, channel-stitched image, and cropped-stitched image - to obtain four moiré detection modules - a scaled image moiré detection module, a cropped image moiré detection module, a channel-stitched image moiré detection module, and a cropped-stitched image moiré detection module. The moiré detection unit is used to process the input image to be determined whether it has moiré to obtain a scaled image, a cropped image, a channel-stitched image, and a cropped-stitched image, and then send the images to the scaled image moiré detection module, the cropped image moiré detection module, the channel-stitched image moiré detection module, and the cropped-stitched image moiré detection module to obtain four probability values, and the four probability values ​​are averaged to obtain a final moiré determination result.

[0015] The technical effect achieved by the present invention is: multiple processing is performed on the original image - scaling, cropping, channel stitching, cropping and stitching, so as to retain various features of the original image as much as possible, such as texture features, and four neural networks are trained separately. Finally, after summarizing, it is determined whether there are moiré patterns in the image, making the moiré pattern detection result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the moiré detection method proposed in the present invention.

[0017] Figure 2 It is a structural schematic diagram of the moiré detection device proposed in the present invention.

[0018] Explanation of the reference numerals in the figure: 1 is a training data production unit, 2 is a neural network training unit, and 3 is a moiré detection unit. DETAILED DESCRIPTION

[0019] See also Figure 1 The moiré detection method proposed in the present invention includes the following steps.

[0020] Step S1: Create a training dataset for a neural network. The training dataset includes four categories: scaled images, cropped images, channel-stitched images, and cropped-stitched images.

[0021] First, a certain number (e.g., 1,000) of images of paper documents or natural scenes are captured or obtained. These images typically do not contain moiré patterns. Also, a certain number (e.g., 1,000) of images of electronic device screens are captured or obtained. These images are collectively referred to as original images. The original images are manually annotated to determine whether they contain moiré patterns.

[0022] The original image is resized to a×b, where a and b are both in the range of 200 pixels to 500 pixels, to obtain a scaled image. Preferably, a=b. The scaled image inherits the moiré annotation of the original image, or is manually reviewed to determine if there is moiré and the annotation is updated.

[0023] The original image is cropped, and only the a×b size of the central area is retained, and the value range of a and b is between 200 pixels and 500 pixels to obtain a cropped image. The cropped image inherits the moiré annotation of the original image, or is manually reviewed again to see if there is moiré and the annotation is updated. Preferably, a=b. If the width of the original image is less than a, the original images are first spliced ​​in the width direction (i.e. horizontally) until the width of the spliced ​​image exceeds a, and then the spliced ​​image is cropped, and only the a×b size of the central area is retained. If the height of the original image is less than b, the original images are first spliced ​​in the height direction (i.e. vertically) until the height of the spliced ​​image exceeds b, and then the spliced ​​image is cropped, and only the a×b size of the central area is retained.

[0024] A color image is a×b in size, with a and b both measured in pixels. Using the three color channels of the RGB color space (R (red), G (green), and B (blue)), this creates a color space with a×b×3 channels. The a×b×3 channels of the scaled image and the a×b×3 channels of the cropped image are superimposed to create a color space with a×b×6 channels, known as a channel-stitched image. The channel-stitched image inherits the moiré annotations of the original image, or is manually reviewed for moiré patterns and updated with annotations.

[0025] The scaled image is cut in half widthwise (i.e., horizontally), either the left or right half, to produce a first intermediate image of (a / 2) × b. The cropped image is cut in half widthwise (i.e., horizontally), either the left or right half, to produce a second intermediate image of (a / 2) × b. The first and second intermediate images are spliced ​​widthwise (i.e., horizontally), either on the left or right side, to produce an a × b image, called a cropped and spliced ​​image. The cropped and spliced ​​image inherits the moiré annotation of the original image, or the annotation is updated by manual review to detect moiré.

[0026] Another method for creating cropped and stitched images is to cut the scaled image in half in the height direction (i.e., vertically), either the top or bottom half is acceptable, to obtain a third intermediate image of size a × (b / 2). Cut the cropped image in half in the height direction (i.e., vertically), either the top or bottom half is acceptable, to obtain a fourth intermediate image of size a × (b / 2). The third and fourth intermediate images are then stitched together in the height direction (i.e., vertically), either the top or bottom half is acceptable, to obtain an image of size a × b, called a cropped and stitched image.

[0027] The training data of traditional neural networks used to detect whether moiré patterns exist in images are usually obtained by reducing the size of the original image, similar to the above-mentioned scaled image. Reducing the image size will lose the texture information of the image, which will cause the detection results of the neural network to be inaccurate. The training data of the neural network used to detect whether moiré patterns exist in images of the present invention are divided into four categories. The cropped image is a small image cut out from the large image. The retained small image is in the center of the large image, which is the focus position and usually contains the most texture information; the image that is cropped is generally the periphery of the large image, which contains only less texture information. Channel stitched images and cropped stitched images are used to increase the diversity of the training data of the neural network using image processing methods of different dimensions.

[0028] Step S2: Using the four categories in the training data set - scaled image, cropped image, channel spliced ​​image, and cropped spliced ​​image - four deep neural networks (DNN) for classification are trained respectively, for example, using lightweight neural networks such as residual networks (ResNet) or MobileNet, to obtain four moiré detection modules respectively - scaled image moiré detection module, cropped image moiré detection module, channel spliced ​​image moiré detection module, and cropped spliced ​​image moiré detection module.

[0029] The training method of the neural network is as follows: input an image of a certain category in the training data set, and output a probability value between 0 and 1, which is used to indicate one of the two categories (there is moiré in the image, there is no moiré in the image), so that the output of the neural network is consistent with the manual annotation of the moiré of the image as much as possible. The neural network trained in this way becomes a moiré detection module, which can be used to detect whether there is moiré in the image. During training, the parameters of the neural network are optimized using the SGD (stochastic gradient descent) or Adam (adaptive moment estimation) algorithm. The last layer of the neural network is replaced by a two-category fully connected layer by an n-category fully connected layer. The two-category fully connected layer outputs a two-dimensional vector. The value of the two-dimensional vector represents the selection of one of the only two categories. The two-dimensional vector is also normalized to a probability value between 0 and 1 by softmax (normalized exponential function) to facilitate subsequent calculations.

[0030] Step S3: For an input image to be determined whether moiré is present, a scaled image, a cropped image, a channel-stitched image, and a cropped-stitched image are first obtained based on the input image. These images are then fed into a scaled image moiré detection module, a cropped image moiré detection module, a channel-stitched image moiré detection module, and a cropped-stitched image moiré detection module, respectively, to obtain four probability values. An ensemble learning method is employed to average the four probability values, and a final moiré determination result is obtained based on the average. For example, an average greater than 0.5 indicates that moiré is present in the image, and an average less than or equal to 0.5 indicates that no moiré is present in the image.

[0031] See also Figure 2 The moiré detection device proposed in the present invention includes a training data production unit 1, a neural network training unit 2, and a moiré detection unit 3. Figure 2 The device shown corresponds to Figure 1 The method shown.

[0032] The training data production unit 1 is used to produce a training data set for a neural network. The training data set includes four categories: scaled images, cropped images, channel spliced ​​images, and cropped spliced ​​images.

[0033] The neural network training unit 2 is used to use the four categories in the training data set - scaled image, cropped image, channel spliced ​​image, and cropped spliced ​​image - to train four deep neural networks for classification, respectively, and obtain four moiré detection modules - scaled image moiré detection module, cropped image moiré detection module, channel spliced ​​image moiré detection module, and cropped spliced ​​image moiré detection module.

[0034] The moiré detection unit 3 is used to process the input image to be judged whether there is moiré to obtain a scaled image, a cropped image, a channel stitched image, and a cropped stitched image, and then send them to the scaled image moiré detection module, the cropped image moiré detection module, the channel stitched image moiré detection module, and the cropped stitched image moiré detection module respectively to obtain four probability values, and the four probability values ​​are averaged to obtain the final moiré judgment result.

[0035] Compared with traditional image moiré detection methods, the present invention has technological innovations in producing training data and training neural networks. It strives to obtain more image features from the original image by fusing multiple features such as changing the image size, cropping the original image, channel splicing, cropping and splicing, and respectively sending them to four neural networks trained with these four types of images. Finally, an integrated learning method is used for judgment, thereby making the moiré detection results more accurate and significantly improving the generalization ability.

[0036] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A moiré detection method, characterized in that: The method includes the following steps: Step S1: Prepare a training dataset for the neural network; the training dataset includes four categories: scaled images, cropped images, channel stitched images, and cropped stitched images; all images are manually annotated to see if they have moiré patterns; First, an original image is captured or obtained, wherein a portion of the original image has moiré patterns, and the remaining original image does not have moiré patterns; Resize the original image to a×b to obtain a scaled image; Crop the original image, retaining only the a×b size of the center area of ​​the original image to obtain a cropped image; The a×b×3 color space channel of the scaled image and the a×b×3 color space channel of the cropped image are superimposed to form a×b×6 color space channel, which is called a channel stitching image; Take half of the scaled image horizontally and half of the cropped image horizontally, and splice the two horizontally to obtain a cropped spliced ​​image; or take half of the scaled image vertically and half of the cropped image vertically, and splice the two vertically to obtain a cropped spliced ​​image; The value range of a and b is between 200 pixels and 500 pixels; Step S2: using the four categories in the training data set - scaled image, cropped image, channel stitching image, and cropped stitching image - to train four deep neural networks for classification respectively to obtain four moiré detection modules respectively - scaled image moiré detection module, cropped image moiré detection module, channel stitching image moiré detection module, and cropped stitching image moiré detection module; Step S3: For the input image to be judged whether there is moiré, first obtain a scaled image, a cropped image, a channel stitched image, and a cropped stitched image based on the input image, and then send them to the scaled image moiré detection module, the cropped image moiré detection module, the channel stitched image moiré detection module, and the cropped stitched image moiré detection module respectively to obtain four probability values, take the average of the four probability values, and obtain the final moiré judgment result based on the average.

2. The moiré detection method according to claim 1, wherein: a=b.

3. The moiré detection method according to claim 1, wherein: The original image is manually annotated to see if there is moiré; the scaled image, cropped image, channel-stitched image, cropped and stitched image, or the moiré annotation of the original image is inherited, or the moiré is manually reviewed again to see if there is moiré and the annotation is updated.

4. The moiré detection method according to claim 1, wherein: If the width of the original image is less than a, the original images are first spliced ​​horizontally until the width of the spliced ​​image exceeds a, and then the spliced ​​image is cropped to retain only the a×b size of the central area to obtain a cropped image; if the height of the original image is less than b, the original images are first spliced ​​vertically until the height of the spliced ​​image exceeds b, and then the spliced ​​image is cropped to retain only the a×b size of the central area to obtain a cropped image.

5. The moiré detection method according to claim 1, wherein: In step S2, the neural network adopts a residual network ResNet or a MobileNet neural network.

6. The moiré detection method according to claim 1, wherein: In step S2, the training method of the neural network is: input an image of a certain category in the training data set, and output a probability value between 0 and 1, which is used to indicate whether there is moiré in the image or there is no moiré in the image, so that the output of the neural network is consistent with the manual annotation of the moiré of the image as much as possible.

7. The moiré detection method according to claim 6, wherein: In step S2, when the neural network is trained, the parameters of the neural network are optimized using the stochastic gradient descent SGD or the adaptive moment estimation Adam algorithm.

8. The moiré detection method according to claim 6, wherein: In step S2, the last layer of the neural network is replaced by a two-category fully connected layer from an n-category fully connected layer. The two-category fully connected layer outputs a two-dimensional vector. The value of the two-dimensional vector represents the selection of one of the only two categories. The two-dimensional vector is also normalized to a probability value between 0 and 1.

9. The moiré detection method according to claim 1, wherein: In step S3, if the average value is greater than 0.5, it indicates that there are moiré patterns in the image; if the average value is less than or equal to 0.5, it indicates that there are no moiré patterns in the image.

10. A moiré detection device, characterized in that: It includes a training data production unit, a neural network training unit, and a moiré detection unit; The training data production unit is used to produce a training data set for a neural network; the training data set includes four categories, namely, scaled images, cropped images, channel spliced ​​images, and cropped spliced ​​images; First, an original image is captured or obtained, wherein a portion of the original image has moiré patterns, and the remaining original image does not have moiré patterns; Resize the original image to a×b to obtain a scaled image; Crop the original image, retaining only the a×b size of the center area of ​​the original image to obtain a cropped image; The a×b×3 color space channel of the scaled image and the a×b×3 color space channel of the cropped image are superimposed to form a×b×6 color space channel, which is called a channel stitching image; Take half of the scaled image horizontally and half of the cropped image horizontally, and splice the two horizontally to obtain a cropped spliced ​​image; or take half of the scaled image vertically and half of the cropped image vertically, and splice the two vertically to obtain a cropped spliced ​​image; The value range of a and b is between 200 pixels and 500 pixels; The neural network training unit is used to use the four categories in the training data set - scaled image, cropped image, channel stitching image, and cropped stitching image - to train four deep neural networks for classification, respectively, to obtain four moiré detection modules - scaled image moiré detection module, cropped image moiré detection module, channel stitching image moiré detection module, and cropped stitching image moiré detection module; The moiré detection unit is used to process the input image to be judged whether there is moiré to obtain a scaled image, a cropped image, a channel stitched image, and a cropped stitched image, and then send them to the scaled image moiré detection module, the cropped image moiré detection module, the channel stitched image moiré detection module, and the cropped stitched image moiré detection module respectively to obtain four probability values, and the four probability values ​​are averaged to obtain the final moiré judgment result.

Images