Image processing method and device, medium, electronic equipment and program product

Abstract

The invention relates to the technical field of image processing, and provides an image processing method and device, a medium, electronic equipment and a program product, and the image processing method comprises the steps: carrying out the noise addition of an original image, and obtaining a noise-added image; determining a similar image similar to the original image from a database; and performing denoising processing on the noise-added image according to the similar image to obtain an output image, the similar image being used for guiding the denoising processing on the noise-added image. Noise adding is firstly carried out, then denoising is carried out in combination with the similar image similar to the original image in the denoising process, due to the fact that additional prior information is provided through the similar image, more accurate recognition and denoising can be helped, and the reference of the similar image enables detail and texture features of the original image to be better reserved in the denoising process. Therefore, the details of the output image are controlled more accurately, and the quality of the output image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to an image processing method, device, medium, electronic device, and program product. Background Art

[0002] Traditional diffusion models work by first adding noise to the original image and then denoising it, ultimately generating a generated image. However, diffusion models are highly dependent on noise, and insufficient training or inappropriate parameters can easily lead to artifacts, blurring, and other poor image quality issues in the generated image. Summary of the Invention

[0003] The purpose of the present disclosure is to provide an image processing method, apparatus, medium, electronic device, and program product to solve the problems in the related art.

[0004] In order to achieve the above object, the present disclosure provides an image processing method, which includes: Noise the original image to obtain a noisy image; Determining a similar image similar to the original image from a database; The noisy image is denoised according to the similar image to obtain an output image, wherein the similar image is used to guide the denoising process performed on the noisy image.

[0005] Optionally, determining a similar image similar to the original image from a database includes: Based on the original image, similar images similar to the original image are determined from a database by at least one of text search, sketch search, and category prompter search.

[0006] Optionally, the database pre-stores a plurality of pre-stored images and at least one tag corresponding to each of the pre-stored images, and based on the original image, determining a similar image similar to the original image from the database through text search includes: Performing image content understanding on the original image to obtain image content text; A similar image similar to the original image is obtained according to the image content text and the labels corresponding to the pre-stored images in the database.

[0007] Optionally, obtaining a similar image similar to the original image based on the image content text and labels corresponding to images pre-stored in the database includes: Matching the image content text with the labels corresponding to the pre-stored images in the database; Counting the number of successful matches between the image content text and the label corresponding to each pre-stored image in the database; The pre-stored image with the largest number of successful label matches is determined as a similar image similar to the original image.

[0008] Optionally, the performing denoising on the noisy image according to the similar image to obtain an output image includes: In the process of iteratively denoising the noisy image, the denoised image predicted by the inverse network each time is compared with the similar image, and the inverse network is adjusted according to the comparison result until a preset denoising effect is achieved to obtain an output image.

[0009] Optionally, the noisy image is a full Gaussian distribution image.

[0010] The present disclosure further provides an image processing device, comprising: The first processing module is configured to perform noise processing on the original image to obtain a noisy image; a second processing module, configured to determine a similar image similar to the original image from a database; The third processing module is configured to perform denoising processing on the noisy image according to the similar image to obtain an output image, wherein the similar image is used to guide the denoising processing on the noisy image.

[0011] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned image processing method when the program is executed by a processor.

[0012] The present disclosure also provides an electronic device, comprising: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the above-mentioned image processing method.

[0013] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0014] Through the above technical solution, noise is first added, and then denoising is performed in combination with similar images similar to the original image during the denoising process. Since similar images provide additional prior information, they can help to more accurately identify and remove noise. In addition, the reference of similar images enables the details and texture features of the original image to be better preserved during the denoising process, thereby making the output image more accurate in controlling the details and improving the quality of the output image.

[0015] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 The figure is a flow chart showing a diffusion model according to an exemplary embodiment.

[0017] Figure 2 The figure is a flow chart showing another diffusion model according to an exemplary embodiment.

[0018] Figure 3 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0019] Figure 4 is a schematic diagram showing an image processing method according to an exemplary embodiment.

[0020] Figure 5 is a flowchart of sub-steps of step S2 according to an exemplary embodiment.

[0021] Figure 6 is a flowchart of sub-steps of step S22 according to an exemplary embodiment.

[0022] Figure 7 is a schematic diagram showing a denoising process according to an exemplary embodiment.

[0023] Figure 8 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0024] Figure 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0026] Traditional diffusion models work by first adding noise to the original image and then denoising it, ultimately generating a generated image. However, diffusion models are highly dependent on noise, and insufficient training or inappropriate parameters can easily lead to artifacts, blurring, and other poor image quality issues in the generated image.

[0027] In order to solve the above problems, we first add noise and then combine it with similar images similar to the original image during the denoising process. Since similar images provide additional prior information, they can help to more accurately identify and remove noise. The reference of similar images enables the details and texture features of the original image to be better preserved during the denoising process, thereby making the output image more accurate in controlling the details and improving the quality of the output image.

[0028] Figure 1 is a flow chart showing a diffusion model according to an exemplary embodiment. Figure 2 is a flow chart showing another diffusion model according to an exemplary embodiment. Figure 1 and Figure 2 , the diffusion model can be carried out by stochastic differential equations, and the stochastic differential equation expression is:

[0029] Among them, dx represents a small change, f is a deterministic change, g is the random part, and w is Brownian motion, which means that random operation is assumed to conform to Brownian motion.

[0030] Convert the above formula into discrete form, when The limit when , the discrete form is:

[0031] is the image state at time t, is the deterministic drift term, which represents the deterministic change of the image state. is the diffusion coefficient, which represents the intensity of the noise, is random noise sampled from a standard normal distribution N(0,I).

[0032] Using the conditional probability method, the conditional probability of the above formula is:

[0033]

[0034] According to the idea of Denoising Diffusion Probabilistic Models (DDPM), Bayesian theorem is used to decompose it into:

[0035]

[0036] It is not difficult to find that when Hours long enough, only when and When very close, It will obviously not be equal to 0, and vice versa, only in this case It will obviously not be equal to 0. Therefore, it is necessary to and The case is close enough to do a myopic analysis, for which we again use the Taylor expansion:

[0037] Get the final loss function:

[0038]

[0039] Here, sθ usually represents a parameterized function of the model, where θ represents the parameters of the diffusion model. This parameterized function is used to predict the denoised version of a given noisy image, that is, to recover a clear image from the noisy image.

[0040] The denoising process is continuously filtered by the loss function, and finally the output image of the diffusion model is obtained.

[0041] The specific process is as follows: 1. Through stochastic differential equations, Gaussian noise is continuously added to guide the image information to completely obey the Gaussian distribution (forward process); 2. Find the conditional probability Expressions of 3. Use the loss function to train the model. Specifically, by solving the minimum value of the loss function, reverse denoising is performed to finally obtain a clear image.

[0042] Figure 3 is a flowchart of an image processing method according to an exemplary embodiment. Figure 4 is a schematic diagram showing an image processing method according to an exemplary embodiment. The image processing method can be applied to electronic devices. Figure 3 and Figure 4 , the image processing method may include steps S1 to S3.

[0043] Step S1: performing noise processing on the original image to obtain a noisy image.

[0044] Noising can involve the addition of Gaussian noise, which adds random perturbations (both in magnitude and direction) to the original image's pixel values, ensuring that the overall image conforms to a Gaussian distribution. This can be achieved by continuously adding Gaussian noise to the original image using a stochastic differential equation (SDE) until the image information completely conforms to the Gaussian distribution, resulting in a fully Gaussian distributed image.

[0045] For example, for a face image, after adding Gaussian noise, the facial contour and facial features will become unclear, and some random spots will appear. As Gaussian noise is continuously added, this effect will become stronger and stronger.

[0046] Step S2: Determine similar images similar to the original image from the database.

[0047] A similar image may be an image that is similar to the original image in terms of content, texture, structure, etc.

[0048] A plurality of pre-stored images are pre-stored in the database, and the plurality of pre-stored images in the database are respectively compared with the original image to determine similar images that are similar to the original image.

[0049] The method for determining similar images can be based on image features, image content, deep learning, etc.

[0050] Feature-based methods extract image features, such as color histograms, texture features (such as gray-level co-occurrence matrices), and shape features. Similar images are then determined by comparing the distance or similarity between these features. For example, for two images with similar textures, their gray-level co-occurrence matrices can be calculated, and the similarity of these features can be compared using methods such as Euclidean distance.

[0051] Content-based methods leverage image content information, such as scene and object categories. Deep learning image classification models can be used to first classify the image and then search for similar images within the same category. For example, for an image of an indoor scene, a classification model can first determine that it is an indoor scene, then search a database for other indoor scene images as similar images.

[0052] Deep learning-based methods use deep learning models such as convolutional neural networks (CNNs) to learn high-level feature representations of images and directly calculate image similarity using the feature vectors output by the models. For example, pre-trained CNN models (such as VGG and ResNet) are used to extract image features, and then methods such as cosine similarity are used to compare feature vectors to identify similar images.

[0053] Step S3: De-noising the noisy image based on the similar images to obtain an output image.

[0054] Among them, the similar images are used to guide the denoising process of the noisy images.

[0055] Similar images can provide information about the texture, edges, etc. that the image should have, helping the denoising algorithm to more accurately distinguish between noise and real image content.

[0056] By first adding noise and then combining it with similar images similar to the original image during the denoising process, similar images provide additional prior information that can help more accurately identify and remove noise. The reference of similar images also enables the details and texture features of the original image to be better preserved during the denoising process, thereby making the output image more precise in controlling details and improving the quality of the output image.

[0057] It should be understood that step S1 may be executed before step S2, or step S2 may be executed before step S1, or step S1 and step S2 may be executed simultaneously, which is not limited in this embodiment.

[0058] In one possible implementation, please refer to Figure 2 , step S2 may include: Based on the original image, similar images similar to the original image are determined from a database through at least one of text search, sketch search, and classification prompter search.

[0059] Optionally, the database pre-stores a plurality of pre-stored images and at least one label corresponding to each pre-stored image. The label may be annotation information of the pre-stored image, and the label may include a subject, an object name, a scene description, an action, etc. For example, an image of a cheetah running may be labeled with labels such as "animal," "cheetah," "running," and "grassland."

[0060] In one embodiment, see Figure 5 Based on the original image, determining a similar image similar to the original image from a database through text search may include step S21 and step S22.

[0061] Step S21: performing image content understanding on the original image to obtain image content text.

[0062] The image recognition technology can be used to understand the content of the original image and generate image text content that can describe the original image. The image text content can include, but is not limited to, themes, object names, scene descriptions, behavioral actions, etc.

[0063] Step S22 , obtaining similar images similar to the original image based on the image content text and the labels corresponding to the images pre-stored in the database.

[0064] In one possible implementation, see Figure 6 , step S22 may include steps S221 to S223.

[0065] Step S221 : Match the image content text with the labels corresponding to the pre-stored images in the database.

[0066] An index is created based on the labels corresponding to pre-stored images in the database to facilitate quick searches. The text in the image content is matched against the labels corresponding to pre-stored images in the database. This matching method can be a similarity calculation (e.g., cosine similarity). If the similarity exceeds a set threshold, the image text content successfully matches the label corresponding to the pre-stored image; otherwise, the match fails.

[0067] Step S222 , counting the number of successful matches between the image content text and the label corresponding to each pre-stored image in the database.

[0068] The image content text may be composed of multiple keywords, and the pre-stored images in the database also correspond to at least one label. Then, multiple keywords are matched with each label corresponding to the pre-stored image respectively to determine the number of successful matches between the image text content and the label corresponding to each pre-stored image.

[0069] Step S223 : Determine the pre-stored image with the largest number of successful label matches as a similar image that is similar to the original image.

[0070] In one embodiment, based on the original image, similar images similar to the original image are determined from the database through sketch search. It can be understood that a sketch is generated according to the content of the original image, and similar images similar to the sketch are determined in the database by analyzing the shape, lines and other features of the sketch.

[0071] For example, sketch feature extraction is first performed. Features can include shape features (such as outline shape) and line features (such as line direction, length, curvature, etc.). For example, if a user draws a simple triangle sketch to represent a mountain, the outline shape and line features of the triangle can be extracted. Then, image feature extraction and indexing are performed. Pre-stored images in the database also need to pre-extract similar features and establish an index. For objects and scenes in pre-stored images, their shapes, lines, and other features can be extracted, such as the outline shape of buildings and the direction of road lines. Finally, feature matching and image retrieval are performed to match the features of the sketch with the features of pre-stored images in the database. Some shape matching algorithms, such as shape context matching and Hough transform, can be used to measure the similarity between the sketch features and the pre-stored image features. The pre-stored image with the greatest degree of similarity is determined to be a similar image similar to the original image.

[0072] Determining similar images based on sketch search can be used in situations where it is difficult to accurately describe the original image with words. It can also be further supplemented based on the text description to make the determined similar images more accurate and precise.

[0073] In one embodiment, based on the original image, a classification prompter is used to identify similar images from a database that are similar to the original image. This can be understood as the classification prompter classifying the original image and searching the database for images belonging to the same or related categories based on the classification results as similar images. The classification prompter can be a pre-trained deep learning classification model that can identify object categories, scene categories, etc. in an image.

[0074] For example, image classification is first performed. The original image is input into the classification prompter, which will output the category to which the original image belongs. For example, for an image of a car, the classification prompter may output the category "car", which can even be further divided into subcategories such as "sedan" and "truck". Category matching and image retrieval are then performed. Based on the classification results, pre-stored images belonging to the same category are searched in the database. If the classification prompter gives multiple possible categories, searches can also be performed within these related categories. For example, if the original image is classified as "sedan", pre-stored images labeled "sedan" are searched in the database as similar images to the original image.

[0075] In one possible implementation, see Figure 7 , step S3 may include: In the process of iterative denoising of the noisy image, the denoised image predicted by the inverse network each time is compared with the similar image, and the inverse network is adjusted according to the comparison result until the preset denoising effect is achieved and the output image is obtained.

[0076] The denoised image predicted by the inverse network is compared with a similar image each time, and the inverse network is adjusted based on the comparison results until the desired denoising effect is achieved, resulting in the output image. Similar images serve as additional supervisory signals to guide the denoising process. By first comparing the denoised image with similar images, an additional loss term is calculated, which measures the difference between the denoised image and the similar image. This additional loss term is then added to the overall loss function, and the model parameters are adjusted with the goal of minimizing the overall loss function.

[0077] Similar images provide additional supervision information, and similar images also help the model better understand the structure and content of the image, thereby improving denoising effects and image quality.

[0078] Building on the aforementioned image processing methods, educational images can be enhanced or directly generated using a sentence, image, or a combination of these methods. This image processing method can also enhance data acquisition autonomy and reduce associated labor costs. By combining this with the capabilities of existing annotation systems, for example, it can manage images generated in batches, thereby generating specialized domain training datasets.

[0079] Based on the same inventive concept, please refer to Figure 8 The present disclosure further provides an image processing device, which can be applied to electronic devices. The image processing device 600 may include: The first processing module 601 is configured to perform noise processing on the original image to obtain a noisy image; The second processing module 602 is configured to determine similar images similar to the original image from the database; The third processing module 603 is configured to perform denoising processing on the noisy image according to the similar image to obtain an output image, wherein the similar image is used to guide the denoising processing on the noisy image.

[0080] Optionally, the second processing module 602 includes: The first sub-processing module is configured to determine, based on the original image, similar images similar to the original image from a database through at least one of text search, sketch search, and classification prompter search.

[0081] Optionally, a database pre-stores a plurality of pre-stored images and at least one tag corresponding to each pre-stored image, and the first sub-processing module includes: A first processing unit is configured to understand the image content of the original image and obtain the image content text; The second processing unit is configured to obtain a similar image similar to the original image according to the image content text and the labels corresponding to the pre-stored images in the database.

[0082] Optionally, the second processing unit is specifically configured to: Match the image content text with the labels corresponding to the pre-stored images in the database; Count the number of successful matches between the image content text and the label corresponding to each pre-stored image in the database; The pre-stored image with the largest number of successful label matches is determined as a similar image that is similar to the original image.

[0083] Optionally, the third processing module 603 is specifically configured to: In the process of iterative denoising of the noisy image, the denoised image predicted by the inverse network each time is compared with the similar image, and the inverse network is adjusted according to the comparison result until the preset denoising effect is achieved and the output image is obtained.

[0084] Optionally, the noisy image is a full Gaussian distribution image.

[0085] Regarding the image processing apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the image processing method, and will not be elaborated here.

[0086] Figure 9 FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. Figure 9 As shown, the electronic device 700 may include: a processor 701 , a memory 702 , and may further include one or more of a multimedia component 703 , an input / output (I / O) interface 704 , and a communication component 705 .

[0087] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned image processing method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, as well as application-related data such as contact information, sent and received messages, images, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0088] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned image processing method.

[0089] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-described image processing method. For example, the computer-readable storage medium may be the aforementioned memory 702 including the program instructions. The program instructions may be executed by the processor 701 of the electronic device 700 to perform the above-described image processing method.

[0090] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a processor, and the computer program implements the steps of the above-mentioned image processing method when executed by the processor.

[0091] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0092] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0093] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Noise the original image to obtain a noisy image; Determining a similar image similar to the original image from a database; The noisy image is denoised according to the similar image to obtain an output image, wherein the similar image is used to guide the denoising process performed on the noisy image.

2. The image processing method according to claim 1, wherein: The determining, from a database, a similar image that is similar to the original image comprises: Based on the original image, similar images similar to the original image are determined from a database through at least one of text search, sketch search, and category prompter search.

3. The image processing method according to claim 2, wherein: The database pre-stores a plurality of pre-stored images and at least one tag corresponding to each of the pre-stored images. Based on the original image, a similar image similar to the original image is determined from the database through text search, including: Performing image content understanding on the original image to obtain image content text; A similar image similar to the original image is obtained according to the image content text and the labels corresponding to the pre-stored images in the database.

4. The image processing method according to claim 3, wherein: The obtaining of a similar image similar to the original image based on the image content text and the labels corresponding to the pre-stored images in the database includes: Matching the image content text with the labels corresponding to the pre-stored images in the database; Counting the number of successful matches between the image content text and the label corresponding to each pre-stored image in the database; The pre-stored image with the largest number of successful label matches is determined as a similar image similar to the original image.

5. The image processing method according to claim 1, wherein: The denoising process is performed on the noisy image according to the similar image to obtain an output image, comprising: In the process of iteratively denoising the noisy image, the denoised image predicted by the inverse network each time is compared with the similar image, and the inverse network is adjusted according to the comparison result until a preset denoising effect is achieved to obtain an output image.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The noisy image is a full Gaussian distribution image.

7. An image processing device, characterized in that: The image processing device comprises: The first processing module is configured to perform noise processing on the original image to obtain a noisy image; a second processing module, configured to determine a similar image similar to the original image from a database; The third processing module is configured to perform denoising processing on the noisy image according to the similar image to obtain an output image, wherein the similar image is used to guide the denoising processing on the noisy image.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the image processing method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

Images