A texture continuity detection method based on self-optimizing depth image prior

Abstract

The application discloses a texture continuity detection method based on a self-optimizing depth image prior, and comprises the following steps: firstly, a negative sample image is generated, a texture feature extractor and an image reconstructor are trained through a self-discriminator; then, a positive sample image and the negative sample image are respectively input into the trained texture feature extraction module and the image reconstruction module, a reconstructed positive sample image and a reconstructed negative sample image are generated, and a residual image corresponding to the positive sample and the negative sample is calculated to train a texture continuity discriminator; finally, an image to be detected is input into the trained texture continuity discriminator for scoring, a corresponding hypothesis probability is calculated, when the hypothesis probability is less than a set threshold, it is judged that the image to be detected is texture continuous, otherwise, it is judged as discontinuous. The application can be used for judging whether an image is spliced or incomplete, and is one of important standards for image quality evaluation.

Description

Technical Field

[0001] The present application belongs to the field of image detection technology, and specifically designs a texture continuity detection method based on self-optimizing depth image prior. Background Art

[0002] With the continuous development of artificial intelligence, image synthesis and fabrication techniques are becoming increasingly realistic. Without a corresponding mechanism to assess the quality of various types of synthesized and fabricated images, significant losses to the social economy and national security are inevitable. For example, in the field of medical imaging, uploaders of medical image data may intentionally or unintentionally upload synthesized, tampered, or erroneous image data. This illegal data wastes resources in medical image data analysis systems, causes program anomalies, and may lead to incorrect diagnostic results. By intentionally uploading this illegal data, uploaders can illegally obtain insurance compensation, higher medical care conditions, or other social benefits. Texture continuity in images is one of the criteria for image quality assessment.

[0003] Texture, a crucial attribute of images, plays a crucial role in computer vision and image processing. Texture often changes relatively smoothly. Discontinuities caused by various reasons appear as unnatural, sudden changes in texture. These irregularities can exhibit significant diversity, locality, and randomness in statistical properties, making them difficult to address using common methods based on the independent and identically distributed (IID) assumption. Furthermore, it's impossible to collect datasets that fully capture all discontinuities. Consequently, many image denoising methods are inapplicable.

[0004] Deep image priors are an unsupervised image reconstruction method widely used in image denoising and enhancement. This method requires no pre-training and only requires a contaminated target image. It leverages the tendency of deep convolutional networks to reconstruct continuous, smooth textures when reconstructing images. Using the target image as the input and expected output for network training, the network iterates to find the optimal image restoration result. By subtracting this restoration result from the input image, discontinuities in the original image can be identified.

[0005] Using traditional deep image prior techniques requires manual selection of intermediate results during the image reconstruction process, choosing the most satisfactory result as the reconstruction output. This process is laborious and subject to user judgment bias. Inappropriate selection can lead to increased iteration time. Furthermore, traditional methods can only be trained on a single image, are unable to train on large amounts of data, and lack universal applicability across datasets. Summary of the Invention

[0006] The purpose of the present application is to propose a texture continuity detection method based on self-optimizing depth image prior, which overcomes the problems of existing depth image prior methods, such as the need for manual intervention, the inability to process large amounts of unlabeled data, poor migration ability, etc.

[0007] In order to achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0008] A texture continuity detection method based on self-optimizing depth image prior, comprising:

[0009] Obtaining a positive sample image, generating a negative sample image with texture discontinuity through image disturbance, inputting the negative sample image into a texture feature extractor to extract negative sample features, then passing the negative sample features through an image reconstructor to obtain a reconstructed negative sample image, calculating a reconstruction loss, training the texture feature extractor and the image reconstructor;

[0010] Inputting the positive sample image and the reconstructed negative sample image into a self-optimizing decision maker for scoring, calculating a self-optimizing loss according to the score, training the self-optimizing decision module, and stopping the training of the texture feature extraction module and the image reconstruction module when the self-optimizing loss converges;

[0011] Inputting the positive sample image and the negative sample image into the trained texture feature extraction module and the image reconstruction module respectively, generating a reconstructed positive sample image and a reconstructed negative sample image, and calculating the corresponding residual images of the positive sample and the negative sample;

[0012] Inputting the corresponding residual images of the positive sample and the negative sample into a texture continuity decision maker for scoring, calculating a difference loss according to the score, and completing the training of the texture continuity decision maker;

[0013] Inputting the image to be detected into the trained texture continuity decision maker for scoring, calculating the corresponding hypothesis probability, and when the hypothesis probability is less than a set threshold, judging that the texture of the image to be detected is continuous, otherwise judging that it is discontinuous.

[0014] Further, the reconstruction loss has the following calculation formula:

[0015] Loss rec =||r y -y||1;

[0016] Wherein, Loss rec is the reconstruction loss function, y represents the negative sample image, r y represents the reconstructed negative sample image r y .

[0017] Further, the self-optimizing loss has the following calculation formula:

[0018]

[0019] wherein, Loss DA is a self-optimization loss function, DA(x) represents a score of a self-optimization decider on a positive sample image x, DA(r y ) represents a score of the self-optimization decider on a reconstructed negative sample image r y , x DA(x) represents a gradient of DA(x).

[0020] Further, the difference loss is calculated according to the following formula:

[0021]

[0022] wherein, Loss diff represents a difference loss function, D(a) represents a score of a texture continuity decider D on a positive sample a, D(b) represents a score of the texture continuity decider D on a negative sample b, and a D(a) represents a gradient of D(a), wherein a represents a residual image corresponding to the positive sample, and b represents a residual image corresponding to the negative sample.

[0023] The present application provides a self-optimization deep image prior technology, which realizes automatic finding of a time point of a best reconstruction result of an image, does not need to manually select and review a reconstructed image, simultaneously trains multiple images in a batch training manner to improve calculation efficiency, and has excellent promotion migration property. The self-optimization deep image prior technology can judge whether an image is continuous in texture, can be used to judge whether an image is spliced or incomplete, and is one of important standards for image quality evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flow chart of a texture continuity detection method based on a self-optimization deep image prior of the present application;

[0025] Figure 2 is a training network of a texture feature extractor and an image reconstructor;

[0026] Figure 3 is a training network of a texture continuity decider. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0028] In one embodiment, as shown in FIG. 1, a texture continuity detection method based on a self-optimization deep image prior is proposed, which comprises the following steps. Figure 1

[0029] ​Step S1, obtaining a positive sample image, generating a negative sample image with texture discontinuity through image disturbance, inputting the negative sample image into a texture feature extractor to extract negative sample features, then passing the negative sample features through an image reconstructor to obtain a reconstructed negative sample image, calculating reconstruction loss, training the texture feature extractor and the image reconstructor.

[0030] In order to train the final texture continuity decision maker, the texture feature extractor and the image reconstruction model are first constructed, and the self-optimizing decision module is used to train

[0031] During training, a positive sample image, i.e., a pre-prepared texture continuous image, is first obtained. The positive sample image x is passed through an image disturber to generate a texture discontinuous image y and defined as a negative sample image. The image size of x and y is C*H*W (where C is the number of channels and H*W is the resolution of the image).

[0032] The image disturber uses the DBSCAN algorithm to cluster and then merge the pixel regions with similar textures in the positive sample image x to form super-pixel blocks. A number of super-pixel blocks are randomly selected and copied and pasted to random positions in the image to construct the negative sample image y with texture discontinuity.

[0033] Then, the constructed texture feature extractor can use a conventional convolutional neural network such as VGG16 or ResNet to perform layer-by-layer downsampling operation on the input negative sample image y. Each layer includes a convolution operation, an average pooling operation and a normalization operation. The image size gradually decreases after each layer operation, and the feature map contains more rich information. The global texture information feature map obtained after five times of downsampling operation is output to obtain the negative sample features.

[0034] The image reconstructor can use the decoder part of the Unet network or other decoder networks to perform layer-by-layer deconvolution operation on the input negative sample features. Each layer includes a deconvolution operation, an average pooling operation and a normalization operation. In the deconvolution operation, the image features are gradually enlarged, and finally restored to the original C*H*W size to obtain the reconstructed negative sample image r y .

[0035] At this time, the reconstruction loss is calculated as:

[0036] Loss rec =||r y -y||1;

[0037] Loss rec is the reconstruction loss function, and the negative sample image y and the reconstructed negative sample image r ythe similarity measure. After the reconstruction loss is calculated, it can be used to backpropagate to update the network parameters of the texture feature extractor and the image reconstructor, and train the texture feature extractor and the image reconstructor.

[0038] With the training of the network, the image reconstructor gradually learns how to restore the previous image. According to the running characteristics of the neural network, there will be a negative sample image that eliminates the phenomenon of discontinuous texture in the restoration process.

[0039] Step S2, input the positive sample image and the reconstructed negative sample image into the self-optimizing discriminator for scoring, calculate the self-optimizing loss according to the score, train the self-optimizing decision module, and stop the training of the texture feature extraction module and the image reconstruction module when the self-optimizing loss converges.

[0040] The embodiment automatically determines the training duration of the texture feature extractor and the image reconstructor through the self-optimizing discriminator.

[0041] The self-optimizing discriminator DA takes the positive sample image x as the positive sample of the self-optimizing discriminator, and the negative sample image r y reconstructed by the image reconstructor as the negative sample of the self-optimizing discriminator. The self-optimizing discriminator is trained synchronously with the texture feature extractor and the image reconstructor.

[0042] In the training process, the self-optimizing loss is calculated according to the score output by the self-optimizing discriminator;

[0043]

[0044] Loss DA is the self-optimizing loss function. When Loss DA is no longer changed, it means that the texture feature extractor and the image reconstructor have reached good results, and the training is completed. DA(x) represents the score of the positive sample image x given by the self-optimizing discriminator, and DA(r y ) represents the score of the reconstructed negative sample r y given by the self-optimizing discriminator, and x DA(x) represents the gradient of DA(x).

[0045] At the beginning of the training, the self-optimizing discriminator gives a large difference score to x and r y respectively. According to the running characteristics of the neural network, the negative sample r y will first approach the positive sample x, and then deviate from x and approach y. In this process, the image reconstructor will definitely produce a negative sample r y that eliminates the phenomenon of discontinuous texture at a certain time T. y When the self-optimizing discriminator is fully trained, it will converge stably, and the score difference of x and r yComparing the similar, this moment is the moment T, the r y is the required negative sample image that eliminates discontinuous texture, and the texture feature extractor and the image reconstructor also meet the training requirements.

[0046] The embodiment automatically judges the training stop point of the texture feature extractor and the image reconstructor according to the score difference of x and r y , without the need for manual judgment as in the traditional method, thereby improving the training efficiency.

[0047] As shown in Figure 2 , G represents the image perturber, V represents the texture feature extractor, E represents the image reconstructor, DA represents the self-optimizing decision maker, and the network shown in Figure 2 is used to train the texture feature extractor and the image reconstructor.

[0048] Step S3: The positive sample image and the negative sample image are respectively input to the trained texture feature extraction module and the image reconstruction module to generate the reconstructed positive sample image and the reconstructed negative sample image, and the residual images corresponding to the positive sample and the negative sample are calculated.

[0049] Next, the texture continuity decision maker is trained. The positive sample image and the negative sample image are respectively input to the trained texture feature extraction module and the image reconstruction module to generate the reconstructed positive sample image and the reconstructed negative sample image, and the residual images corresponding to the positive sample and the negative sample are calculated.

[0050] That is, after the texture feature extractor and the image reconstructor are trained, the positive sample x and the negative sample y are used to generate the reconstructed positive sample r x and the reconstructed negative sample r y by the trained texture feature extractor and the image reconstructor, thereby obtaining two pairs of residual images x-r x and y-r y .

[0051] Step S4: The residual images corresponding to the positive sample and the negative sample are input to the texture continuity decision maker for scoring, and the difference loss is calculated according to the score, thereby completing the training of the texture continuity decision maker.

[0052] In the training of the texture continuity decision maker D, the loss function used in the embodiment is as follows:

[0053]

[0054] wherein Loss diff represents the difference loss function, D(a) represents the score of the positive sample a given by the texture continuity decision maker D, D(b) represents the score of the negative sample b given by the texture continuity decision maker D, and ▽ aD(a) represents the gradient of D(a), where a represents the residual image corresponding to the positive sample, and b represents the residual image corresponding to the negative sample. ||.||1 and ||.||2 represent the L1 norm and the L2 norm, respectively.

[0055] The second term on the right side of the loss function equation is a regularization penalty term, which can make the texture continuity discriminator form a relatively flat hyperplane near the positive sample. On the one hand, it requires the texture continuity discriminator D to give a similar score to the positive sample for similar data, so that the score of the positive sample by the texture continuity discriminator D has high cohesion. On the other hand, it can regularize the texture continuity discriminator to prevent it from diverging.

[0056] The texture continuity discriminator D described in this embodiment will y-r y As a negative sample of the discriminator, y-r y The discontinuous texture in the negative sample is retained. In order to be symmetrical, x-r x As a positive sample of the discriminator, x-r x The reconstruction error is extremely small, ideally 0. The texture continuity discriminator D will give a score for the positive sample and the negative sample, respectively. During training, the difference between the scores of the positive sample and the negative sample by the texture continuity discriminator D is as large as possible, while the regularization prevents the difference from being too large and unable to converge. After iterative training, the texture continuity discriminator can give a similar score to the positive sample for similar data, and the scores of the positive sample and the negative sample are reasonably distributed, without causing excessive differentiation of the scores at both ends.

[0057] As Figure 3 shown, G represents an image perturber, V represents a texture feature extractor, E represents an image reconstructor, and D represents a texture continuity discriminator. The network shown in Figure 3 is used to train the texture continuity discriminator.

[0058] Step S5, input the image to be detected into the trained texture continuity discriminator for scoring, calculate the corresponding hypothesis probability, and when the hypothesis probability is less than a set threshold, judge that the texture of the image to be detected is continuous, otherwise, judge that it is not continuous.

[0059] In this embodiment, the image to be detected is input into the texture continuity discriminator D for scoring, and then the corresponding hypothesis probability (p-value) is calculated. Generally, when the p-value is less than 0.05, it is considered that the score value does not conform to the given probability distribution, so that the input image is determined not to conform to the texture continuity. The probability density function used for scoring adopts a normal distribution with a mean of 0 and a variance of 1.

[0060] It should be noted that the texture feature extractor, the image reconstructor, the self-optimizing discriminator, and the texture continuity discriminator described in the present application can be realized by a neural network, and a multi-layer convolutional neural network is a relatively optimal implementation method.

[0061] The beneficial effects of the present application mainly manifest in: 1. Combining the depth image prior technology and the self-optimizing decision maker, the training time point of eliminating the discontinuous texture image reconstruction can be automatically found, and the efficiency is improved in the batch training mode. 2. The texture feature extractor and the image reconstructor can strip out the features that have eliminated or suppressed the discontinuous texture after training, which are used for the training of the texture continuity decision maker.

[0062] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for texture continuity detection based on self-optimizing depth image priors, characterized in that, The texture continuity detection method based on a self-optimizing depth image prior, comprising: Obtaining a positive sample image, generating a negative sample image with texture discontinuity through image disturbance, inputting the negative sample image into a texture feature extractor to extract negative sample features, then inputting the negative sample features into an image reconstructor to obtain a reconstructed negative sample image, calculating a reconstruction loss, training the texture feature extractor and the image reconstructor; Inputting the positive sample image and the reconstructed negative sample image into a self-optimizing decision maker for scoring, calculating a self-optimizing loss according to the score, training the self-optimizing decision module, and stopping the training of the texture feature extractor and the image reconstructor when the self-optimizing loss converges; Inputting the positive sample image and the negative sample image into the trained texture feature extractor and the image reconstructor respectively, generating a reconstructed positive sample image and a reconstructed negative sample image, calculating a difference between the positive sample image and the reconstructed positive sample image to obtain a residual image corresponding to the positive sample, and calculating a difference between the negative sample image and the reconstructed negative sample image to obtain a residual image corresponding to the negative sample; Inputting the residual images corresponding to the positive sample and the negative sample into a texture continuity decision maker for scoring, calculating a difference loss according to the score, and completing the training of the texture continuity decision maker; Inputting an image to be detected into the trained texture continuity decision maker for scoring, calculating a corresponding hypothesis probability, and judging that the image to be detected has texture discontinuity when the hypothesis probability is less than a set threshold, otherwise judging that the image to be detected has texture continuity; The reconstruction loss has the following calculation formula: ； wherein, is a reconstruction loss function, y denotes a negative sample image, denotes a reconstructed negative sample image.

2. The texture continuity detection method based on self-optimizing depth image priors according to claim 1, characterized in that, The self-optimizing loss has the following calculation formula: ； wherein, is a self-optimizing loss function, denotes the score of the self-optimizing decider on a positive sample image x, denotes the score of the self-optimizing decider on a reconstructed negative sample image , denotes the gradient of .

3. The texture continuity detection method based on self-optimizing depth image priors according to claim 1, characterized in that, The difference loss has the following calculation formula: ； wherein, denotes the difference loss function, denotes the score of the texture continuity decider D on positive samples , denotes the score of the texture continuity decider D on negative samples b , denotes the gradient of , wherein denotes the residual image corresponding to the positive sample, b denotes the residual image corresponding to the negative sample.

Images