Method and device for removing image noisy points and storage medium
By acquiring the potential spatial vectors and super-segment processing of the noise image, combined with the VAE encoder and LLM model, the problem of poor image noise repair effect in the prior art is solved, and a more efficient image denoising effect is achieved.
Patent Information
- Application Number
- CN202510359888.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-25
- Publication Date
- 2025-07-11
AI Technical Summary
In the prior art, image noise repair effect is poor, especially when amplifying video images, the commonly used denoising technology has significant disadvantages.
By acquiring the potential spatial vector of the noise image, determining the synthetic image, and super-segment processing is performed, combining the VAE encoder and the LLM model, noise is gradually removed, including obtaining image text description, gap image recovery and super-segment image expansion, forming a denoising processing of multi-directional information sources.
Improves the effect of image noise repair and enhances image clarity and detail fidelity.
Smart Images
Figure CN120298240A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computers, and in particular, to a method, device, and storage medium for removing image noise. Background Art
[0002] During the acquisition and formation of images, it is inevitable to be interfered by noise. Moreover, the noise in some images is very serious. The noise in the image often interweaves with the signal, making the details of the image itself, such as the boundary contours, lines, etc., become blurred. Image denoising is a widely used technology in image preprocessing, and its purpose is to improve the signal-to-noise ratio of the image and highlight the desired features of the image. Existing methods usually use the encoder-decoder architecture design and only utilize the latent space, resulting in poor repair effects of image noise and not very fine noise removal. For enlarged video images, since the number of noise pixels to be removed significantly increases after enlargement, the disadvantages of applying common denoising technologies become more obvious. Summary of the Invention
[0003] An object of this application is to provide a method, device, and storage medium for removing image noise, so as to solve the problem of poor repair effects of image noise in the prior art.
[0004] According to one aspect of this application, a method for removing image noise is provided. The method includes:
[0005] Obtain the latent space vector of the noisy image, and determine a synthetic image according to the noisy image;
[0006] Obtain a roughly denoised image according to the latent space vector and the synthetic image;
[0007] Perform super-resolution processing on the synthetic image to obtain a super-resolution image;
[0008] Obtain a denoised image according to the synthetic image, the super-resolution image, and the roughly denoised image.
[0009] Optionally, determining a synthetic image according to the noisy image includes:
[0010] Obtain the image text description in the noisy image, and input the image text description into the text-to-image model to output a synthetic image.
[0011] Optionally, obtaining a roughly denoised image according to the latent space vector and the synthetic image includes:
[0012] Determine a difference image according to the latent space vector and the synthetic image;
[0013] Restore the latent space vector to obtain a restored image;
[0014] Obtain a roughly denoised image according to the difference image and the restored image.
[0015] Optionally, determining a difference image based on the latent space vector and the synthesized image includes:
[0016] Obtaining the synthesized latent space vector of the synthesized image;
[0017] Subtracting the synthesized latent space vector from the latent space vector to obtain a difference vector;
[0018] Passing the difference vector through a VAE decoder to obtain a difference image.
[0019] Optionally, obtaining a denoised image based on the synthesized image, the super-resolved image, and the roughly denoised image includes:
[0020] Inputting the synthesized image, the super-resolved image, and the roughly denoised image into a VAE encoder to synthesize a matrix and inputting the matrix into an LLM model;
[0021] Obtaining the denoised image according to the output matrix of the LLM model.
[0022] Optionally, obtaining a denoised image according to the output matrix of the LLM model includes:
[0023] Taking the vector corresponding to the position of the roughly denoised image in the output matrix of the LLM model to obtain a denoised matrix;
[0024] Passing the denoised matrix through a VAE decoder to obtain a denoised image.
[0025] Optionally, performing super-resolution processing on the synthesized image to obtain a super-resolved image, including:
[0026] Inputting the synthesized image into a super-resolution model and enlarging the synthesized image according to the size multiple supported by the super-resolution model to obtain a super-resolved image.
[0027] According to another aspect of the present application, there is also provided a device for removing image noise, the device includes:
[0028] One or more processors; and
[0029] A memory storing computer-readable instructions, the computer-readable instructions, when executed, cause the processor to perform the operations of the method as described above.
[0030] According to still another aspect of the present application, there is also provided a computer-readable storage medium, on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement the method as described above.
[0031] Compared with the prior art, the present application determines a synthetic image according to a noisy image by obtaining a latent space vector of the noisy image; obtains a roughly denoised image according to the latent space vector and the synthetic image; performs super-resolution processing on the synthetic image to obtain a super-resolution image; and obtains a denoised image according to the synthetic image, the super-resolution image, and the roughly denoised image. Thus, the information sources of the image can be used in multiple directions to improve the noise repair effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Other features, objects, and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:
[0033] Figure 1 FIG. shows a schematic flowchart of a method for removing image noise provided according to an aspect of the present application;
[0034] Figure 2 FIG. shows a schematic flowchart of image noise repair in a specific embodiment of the present application.
[0035] Identical or similar reference numerals in the drawings represent identical or similar components. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0036] The present application will be further described in detail below with reference to the drawings.
[0037] In a typical configuration of the present application, a terminal, a device of a service network, and a trusted party each include one or more processors (e.g., a central processing unit (CPU)), an input / output interface, a network interface, and a memory.
[0038] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0039] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, Phase-Change RAM (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technologies, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media such as modulated data signals and carrier waves.
[0040] Figure 1 FIG. shows a schematic flowchart of a method for removing image noise provided according to an aspect of the present application. The method includes: steps S11 to S14, where
[0041] Step S11: Obtain the latent space vector of the noisy image and determine the synthesized image according to the noisy image. Here, the noisy image is the image that needs to be repaired for noise, that is, the noise in the image needs to be removed. The noisy image is input into the encoder of the VAE. The encoder is responsible for mapping the input noisy image to a low-dimensional latent space vector, so that the latent space vector of the noisy image can be obtained. On the one hand, obtain the latent space vector of the noisy image, and on the other hand, use the noisy image to synthesize the required synthesized image. The semantic content contained in the synthesized image is consistent with the text description of the noisy image, so as to more accurately locate the noise and perform precise denoising in the subsequent process.
[0042] Step S12: Obtain a rough denoised image based on the latent space vector and the synthetic image. Here, after obtaining the latent space vector of the noisy image and the image required for synthesis, the two can be used to perform preliminary denoising processing, compare the corresponding vectors, and process the image using the difference to obtain a rough denoised image, that is, the image after preliminary denoising. This image still requires further denoising.
[0043] Step S13: Perform super-resolution processing on the synthetic image to obtain a super-resolved image. Here, perform super-resolution processing on the synthetic image that describes the semantics of the text of the noisy image. The super-resolution processing here refers to an enlarged processing of the image, that is, the size is increased by a multiple, and the image content will be clearer. Through the generated super-resolved image, the noise points can be located more accurately and efficiently.
[0044] Step S14: Obtain a denoised image based on the synthetic image, the super-resolved image, and the rough denoised image. Here, according to the latent space vector, the synthetic image, and multi-faceted means or information sources, improve the restoration effect to obtain a denoised image; specifically, use the synthetic image, the latent space vector, the rough denoised image obtained from the synthetic image, and the super-resolved image obtained from the synthetic image to perform noise point restoration on the noisy image.
[0045] In some embodiments of the present application, in step S11, obtain the image text description in the noisy image, and input the image text description into a text-to-image model to output a synthetic image.
[0046] Use the Blip2 model to obtain the image text description of the noisy image, and then input the obtained image text description into an open-source text-to-image model. The text description is generally "what thing or who did what thing in where"; among them, the text-to-image model can, for example, adopt a pre-trained stablediffusion model with a UNet architecture to output a synthetic image associated with the text description. Among them, the Blip2 model is a model after visual language pre-training, and it realizes open multi-modal content understanding and generation in multi-modal tasks.
[0047] In some embodiments of the present application, in step S12, determine a difference image according to the latent space vector and the synthetic image; restore the latent space vector to obtain a restored image; obtain a rough denoised image according to the difference image and the restored image. Here, when determining the rough denoised image, the vector of the synthetic image can be compared with the obtained latent space vector to obtain a difference image, and then the latent space vector is restored again. Thus, according to the difference between the restored image and the difference image, a rough denoised image can be obtained. When restoring the latent space vector, it is performed through the decoder of the VAE to obtain the image again.
[0048] Specifically, obtain the synthetic latent space vector of the synthetic image; subtract the synthetic latent space vector from the latent space vector to obtain a difference vector; pass the difference vector through the VAE decoder to obtain a difference image. Here, the synthetic latent space vector is obtained by passing the synthetic image through the VAE encoder. Since the semantic content of the noisy image and the synthetic image is the same, the latent space vectors obtained by passing through the VAE encoder have the same distribution. Therefore, subtracting the synthetic latent space vector from the latent space vector gives a difference vector, which represents the difference in the vector space. Passing the difference vector through the VAE decoder gives a difference image, which represents the difference in the pixel space. At the same time, the latent space vector passes through the VAE decoder to obtain a restored image. Subtracting the difference image from the restored image can obtain a roughly denoised image. It should be noted that to ensure that the information obtained is a positive gain, the synthetic latent space vector needs to be subtracted from the latent space vector, rather than subtracting in the reverse direction.
[0049] In some embodiments of the present application, in step S13, the synthetic image is input into the super-resolution model, and the synthetic image is enlarged according to the size multiple supported by the super-resolution model to obtain a super-resolution image. Here, the super-resolution image is exactly the same as the content of the synthetic image, but the size becomes larger and the content is clearer. The synthetic image is a low-resolution image. After being input into the super-resolution model, a high-resolution image is obtained. The size multiple of the high-resolution image is the multiple supported by the selected super-resolution model, and the image is enlarged. For example, if the super-resolution model supports 4 times, then both the length and width of the image are enlarged by four times. The low-resolution image is super-resolved, making it easier for users to intuitively view the image content and the image is clearer.
[0050] In some embodiments of the present application, in step S14, the synthetic image, the super-resolution image, and the roughly denoised image are all input into the VAE encoder to synthesize a matrix, which is then input into the LLM model; the denoised image is obtained according to the output matrix of the LLM model. Here, the synthetic image is super-resolved to obtain the super-resolution image. Both the synthetic image and the super-resolution image pass through a VAE encoder, and the roughly denoised image also passes through this VAE encoder. The three are combined into a matrix and input into the LLM model, and the final denoised image is obtained using the matrix output by the LLM model.
[0051] Specifically, the process of obtaining the denoised image through the LLM model is as follows: Take the vectors corresponding to the positions of the roughly denoised images in the output matrix of the LLM model to obtain the denoising matrix; Pass the denoising matrix through the VAE decoder to obtain the denoised image. Here, take the vectors corresponding to the positions of the roughly denoised images in the output matrix of the LLM model, and map these vectors back to images through the VAE decoder, that is, the final denoised image is obtained. For example, the features of the synthesized image are (N, 512), the features of the super-resolved image are (N*16, 512), the height and width are both 4 times that of the synthesized image, a total of 16 times; the features of the roughly denoised image are (N, 512), then the synthesized image, the super-resolved image, and the roughly denoised image are used as the input of the LLM model after passing through the VAE encoder as (N*18, 512), and the output is (N*18, 512). Then, in the output matrix, the vectors corresponding to the positions of the roughly denoised images, that is, the extracted vectors form (N, 512), and these vectors are passed through the VAE decoder to obtain the denoised image.
[0052] Figure 2 The flowchart of image noise repair in a specific embodiment of the present application is shown. The specific process is as follows: Input the noisy image, and use the VAE encoder to obtain the latent space vector; For the noisy image, use Blip2 to obtain the image text description, and then input the image text description into stable diffusion to output the synthesized image; The synthesized image passes through the VAE encoder to obtain the synthesized latent space vector; Subtract the synthesized latent space vector from the latent space vector to obtain the difference vector; The difference vector passes through the VAE decoder to obtain the difference image; At the same time, the latent space vector passes through the VAE decoder to obtain the restored image; Subtract the difference image from the restored image to obtain the roughly denoised image; The synthesized image is super-resolved to obtain the super-resolved image. The synthesized image, the super-resolved image, and the roughly denoised image all pass through the VAE encoder and are synthesized into a matrix and input into the LLM model; Take the vectors corresponding to the positions of the roughly denoised images in the output matrix of the LLM model, and then pass through the VAE decoder to obtain the denoised image. Furthermore, the information sources of the image can be used in multiple aspects to improve the noise repair effect.
[0053] In addition, an embodiment of the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement the foregoing method for removing image noise.
[0054] Corresponding to the method described above, the present application also provides a terminal, which includes the ability to execute the above Figure 1 or Figure 2modules or units of the method steps described in each embodiment. These modules or units can be implemented by hardware, software, or a combination of both, and this application does not limit it. For example, in an embodiment of this application, a device for removing image noise is further provided. The device includes:
[0055] one or more processors; and
[0056] a memory storing computer-readable instructions, and the computer-readable instructions, when executed, cause the processor to perform the operations of the method as described above.
[0057] For example, when the computer-readable instructions are executed, they cause the one or more processors to:
[0058] obtain the latent space vector of the noisy image, and determine the synthetic image according to the noisy image;
[0059] obtain a roughly denoised image according to the latent space vector and the synthetic image;
[0060] perform super-resolution processing on the synthetic image to obtain a super-resolution image;
[0061] obtain a denoised image according to the synthetic image, the super-resolution image, and the roughly denoised image.
[0062] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these changes and modifications.
[0063] It should be noted that this application can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and similar devices. Additionally, some steps or functions of this application can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.
[0064] In addition, a part of the present application can be applied as a computer program product, for example, computer program instructions. When executed by a computer, through the operation of the computer, it can call or provide the methods and / or technical solutions according to the present application. The program instructions for calling the methods of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in the working memory of a computer device that runs according to the program instructions. Here, an embodiment according to the present application includes a device, the device includes a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, the device is triggered to run based on the methods and / or technical solutions according to the foregoing multiple embodiments of the present application.
[0065] For those skilled in the art, it is obvious that the present application is not limited to the details of the above-mentioned exemplary embodiments, and without departing from the spirit or basic characteristics of the present application, the present application can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be construed as limiting the claimed rights. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. First, second, etc. are used to denote names and do not denote any particular order.
Claims
1. A method for removing image noise, characterized in that, The method includes: Obtain the latent space vector of the noisy image, and determine the synthetic image according to the noisy image; Obtain the roughly denoised image according to the latent space vector and the synthetic image; Perform super-resolution processing on the synthetic image to obtain the super-resolution image; Obtain the denoised image according to the synthetic image, the super-resolution image, and the roughly denoised image.
2. The method according to claim 1, characterized in that, Determining the synthetic image according to the noisy image includes: Obtain the image text description in the noisy image, and input the image text description into the text-to-image model to output the synthetic image.
3. The method according to claim 1, wherein Obtaining the roughly denoised image according to the latent space vector and the synthetic image includes: Determine the difference image according to the latent space vector and the synthetic image; Restore the latent space vector to obtain the restored image; Obtain the roughly denoised image according to the difference image and the restored image.
4. The method according to claim 3, wherein Determining the difference image according to the latent space vector and the synthetic image includes: Obtain the synthetic latent space vector of the synthetic image; Subtract the synthetic latent space vector from the latent space vector to obtain the difference vector; Pass the difference vector through the VAE decoder to obtain the difference image.
5. The method according to claim 1, characterized in that Obtaining the denoised image according to the synthetic image, the super-resolution image, and the roughly denoised image includes: Input the synthetic image, the super-resolution image, and the roughly denoised image into the VAE encoder to synthesize a matrix, and input it into the LLM model; Obtain the denoised image according to the output matrix of the LLM model.
6. The method according to claim 5, wherein Obtaining the denoised image according to the output matrix of the LLM model includes: Take the vector corresponding to the position of the roughly denoised image in the output matrix of the LLM model to obtain the denoising matrix; Pass the denoising matrix through the VAE decoder to obtain the denoised image.
7. The method according to claim 1, wherein Performing super-resolution processing on the synthetic image to obtain the super-resolution image includes: Input the synthetic image into the super-resolution model, and expand the synthetic image according to the size multiple supported by the super-resolution model to obtain the super-resolution image.
8. An apparatus for removing image noise, characterized in that, The device includes: One or more processors; and A memory storing computer-readable instructions that, when executed, cause the processor to perform the operations of the method according to any one of claims 1 to 7.
9. A computer-readable storage medium having stored thereon computer-readable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 7.