Image processing method and related equipment

Abstract

The invention provides an image processing method and related equipment. The method comprises the following steps: determining a target area of an image; in response to an amplification instruction for the target area, performing at least two times of magnification on the target area to obtain a first image; performing noise processing on the first image to obtain a second image; and generating a target image according to the second image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and related equipment. Background Art

[0002] In the field of image processing, one research direction is how to generate high-resolution images from low-resolution images. For example, for an image, how to still display a clear image when it is magnified many times.

[0003] However, the inventors of the present disclosure have found that the high-resolution images generated in the related art do not produce good results. Summary of the invention

[0004] The present disclosure proposes an image processing method and related devices to solve or partially solve the above problems.

[0005] In a first aspect, the present disclosure provides an image processing method, comprising:

[0006] Determine the target area of ​​the image;

[0007] In response to a zoom-in instruction for the target area, the target area is zoomed in at least twice to obtain a first image;

[0008] Performing noise processing on the first image to obtain a second image;

[0009] A target image is generated according to the second image.

[0010] In a second aspect of the present disclosure, there is provided an image processing device, comprising:

[0011] The determination module is configured to: determine a target area of ​​the image;

[0012] an enlargement module, configured to: in response to an enlargement instruction for the target area, enlarge the target area at least twice to obtain a first image;

[0013] A noise module is configured to: perform noise processing on the first image to obtain a second image;

[0014] The generating module is configured to generate a target image according to the second image.

[0015] In a third aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0016] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0017] The image processing method and related equipment provided by the embodiments of the present disclosure can obtain a clearer high-resolution image that is magnified more times by magnifying the target area by at least two times and then adding noise, thereby generating a target image based on the target area that has been magnified at least twice and added with noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram showing an exemplary system provided by an embodiment of the present disclosure is shown.

[0020] Figure 2A A flowchart of an exemplary method provided by an embodiment of the present disclosure is shown.

[0021] Figure 2B A flowchart of an exemplary method for generating a target image according to an embodiment of the present disclosure is shown.

[0022] Figure 3A A schematic diagram showing an exemplary image according to an embodiment of the present disclosure is shown.

[0023] Figure 3B The embodiment of the present disclosure is shown Figure 3A Schematic diagram of an exemplary image after zooming in on the target area.

[0024] Figure 3C Shows the Figure 3A A schematic diagram of an exemplary image after a magnification of a target area.

[0025] Figure 4A A schematic diagram of an exemplary model provided by an embodiment of the present disclosure is shown.

[0026] Figure 4B A schematic diagram is shown of an exemplary process of adding noise to a first image according to an embodiment of the present disclosure.

[0027] Figure 5 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of the present disclosure is shown.

[0028] Figure 6 A schematic diagram of an exemplary device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0032] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0035] Figure 1 A schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure is shown.

[0036] like Figure 1 As shown, the system 100 may include a terminal device 102, a terminal device 104, a server 106, and a database server 108. A medium (e.g., a network) providing a communication link may be included between the terminal device 102, the terminal device 104 and the server 106 and the database server 108. The network may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0037] Various applications (APPs) may be installed on the terminal device 104, such as image processing applications, video conferencing applications, reading applications, video applications, social applications, payment applications, web browsers, and instant messaging tools, etc. These applications may be used to display the delivered information.

[0038] The terminal device 102 and the terminal device 104 here can be hardware or software. When the terminal device 102 and the terminal device 104 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players, laptop computers (Laptops) and desktop computers (PCs), etc. When the terminal device 102 and the terminal device 104 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0039] The server 106 may be a server that provides various services, such as a background server that provides support for various applications displayed on the terminal devices 102 and 104. The database server 108 may also be a database server that provides various services. It is understood that if the server 106 can implement the relevant functions of the database server 108, the database server 108 may not be set in the system 100.

[0040] Here, the server 106 and the database server 108 can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0041] It should be noted that the image processing method provided in the embodiment of the present disclosure may be executed by the terminal device 102 and / or the terminal device 104. It should be understood that Figure 1 The number of terminal devices, users, servers and database servers in the embodiment is only for illustration. Any number of terminal devices, users, servers and database servers may be provided as required.

[0042] In one embodiment, an image processing application or software may be installed in the terminal device 104, and the user 112 may use the application or software to process the image, for example, to enlarge the image and display the enlarged image.

[0043] In some embodiments, the system 100 may be used to train a machine learning model for processing an image. For example, the user 110 may deploy a machine learning model to be trained in the server 106 and may use the terminal device 102 to design training samples, and then use the training samples to train the machine learning model, and during the training process, may use the terminal device 102 to adjust the parameters of the machine learning model until the machine learning model is trained.

[0044] Optionally, the pre-trained machine learning model can be deployed in the server 106. In this case, when the terminal device 104 processes an image, it can upload the image to the server 106 for processing, and the server 106 returns the processed image to the terminal device 104 for display. In some cases, if the pre-trained machine learning model can be lightweight, it can be deployed in the terminal device 104. When processing an image, the terminal device 104 can call the machine learning model stored locally to process the image without processing through the server 106.

[0045] The inventors of the present disclosure have found that, in the related art, although many methods are used to improve the resolution of the enlarged image, the magnification is limited. For example, most methods can only achieve ×4 times super-resolution (only when the image is enlarged by 4 times to present a good resolution effect), and there is still no suitable solution for super-resolution with a larger magnification (for example, ×16 times super-resolution) or infinite magnification super-resolution (referring to the user arbitrarily specifying an area in the image and achieving clarity).

[0046] In view of this, an embodiment of the present disclosure provides an image processing method, which can obtain a clearer high-resolution image that is magnified more times by magnifying a target area by at least two times, then adding noise, and then generating a target image based on the target area that has been magnified at least twice and added with noise.

[0047] Figure 2A FIG. 2 is a flow chart of an exemplary method 200 provided in an embodiment of the present disclosure. The method 200 may be used to process an image. Optionally, the method 200 may be performed by Figure 1The terminal devices 102 and 104 are implemented separately, or can be implemented by Figure 1 The method 200 is implemented by the system 100. The following describes the method 200 by using the terminal device 104.

[0048] like Figure 2A As shown, the method 200 may further include the following steps.

[0049] In step 202, a target area of ​​the image is determined.

[0050] Figure 3A A schematic diagram of an exemplary image 300 according to an embodiment of the present disclosure is shown.

[0051] like Figure 3A As shown, the image 300 is an image with a general resolution (e.g., 1080×768). The image 300 can be displayed on the display screen of the terminal device 104, and the user 112 can select a target area 302 (which can also be considered as a low-quality image LR or a low-resolution image) to be enlarged in the image 300 by touching or other means.

[0052] In step 204, in response to the zoom-in instruction for the target area 302, the target area 302 is zoomed in at least twice to obtain a first image. Optionally, the zoom-in instruction may be, for example, triggered by double-clicking or long pressing the target area. In some embodiments, the zoom-in instruction may be triggered by stretching two fingers in opposite directions.

[0053] In this embodiment, in order to obtain a target image with a higher resolution than the image block corresponding to the target area, the target area is firstly enlarged in this step. In order to obtain a clearer high-resolution image that is enlarged more times, the target area can be enlarged at least twice in this step.

[0054] In some embodiments, in order to obtain a high-resolution image with better effect, a machine learning model may be used to perform image processing.

[0055] Figure 4A A schematic diagram of an exemplary model 400 provided by an embodiment of the present disclosure is shown.

[0056] like Figure 4A As shown, the model 400 may include a pre-trained first model 402 and a second model 404, which are used to enlarge the target area 302 and the target area 302 that has been enlarged once (ie, the first intermediate image 4082), respectively. Figure 4AAs shown, the model 400 may also include a pre-trained third model 406 for generating a target image 4086 based on the noise-processed first image 4084 (ie, the second image).

[0057] As mentioned above, after the model 400 is trained, it can be deployed in the server 106 or the terminal device 104 according to the actual situation and different requirements.

[0058] In some embodiments, in order to obtain a super-resolution image (Super-Resolution, referred to as SR), the first model 402 and the second model 404 can use a super-resolution model (i.e., a super-resolution model) to enlarge the target area 302 and the first intermediate image 4082, so as to obtain a target image with a higher resolution. It can be understood that the super-resolution model can be any model that can achieve super-resolution processing, for example, a convolutional neural network (CNN), a generative adversarial network (GAN), a variational auto-encoder (VAE), a normalization flow model (NF), a diffusion model (DM), and the like.

[0059] In some embodiments, reference Figure 4A As shown, the target area is magnified at least twice to obtain a first image, including:

[0060] Using a pre-trained first diffusion model (first model 402), the target region 302 is magnified by multiples to obtain a first intermediate image 4082 (for example, ×4SR), wherein the target region 302 can be used as a condition of the first diffusion model to constrain the output of the first diffusion model;

[0061] The first intermediate image 4082 is magnified by a multiple using a pre-trained second diffusion model (second model 404) to obtain the first image 4084 (for example, ×16SR), wherein the first intermediate image 4082 can be used as a condition of the second diffusion model to constrain the output of the second diffusion model.

[0062] The inventors of the present disclosure have found that, compared with the high-resolution images that can be obtained by using any single super-resolution model, the diffusion model in the related art has not made any breakthrough in the application scenario. However, when two diffusion models are cascaded and the image is magnified by multiples, a high-resolution image (or super-resolution image) with a larger multiple can be obtained, and significant detail improvement can be obtained. In addition, experiments have shown that the scheme of cascading two networks can effectively improve the effect of super-resolution magnification, and the result of ×16SR is more realistic than that of ×4SR.

[0063] In step 206, noise processing is performed on the first image to obtain a second image.

[0064] In this step, in order to ensure the consistency between the output of the diffusion model and the input image content, the first image 4084 (eg, ×16SR) is inverted, that is, noise 4102 is added to the first image 4084 .

[0065] In some embodiments, noise processing is performed on the first image to obtain the second image, including: adding noise to the first image 4084 multiple times to obtain the second image, thereby better aligning the output and input image contents.

[0066] Optionally, the noise 4102 may be Gaussian noise. Furthermore, the first image 4084 may be subjected to t times of noise addition process, wherein the first image 4084 is set to x0, and the second image obtained after t times of noise addition process is set to xt, and the noise addition process is related to the random Gaussian noise ε and the fusion parameter α.

[0067] Figure 4B A schematic diagram is shown of an exemplary process of adding noise to a first image according to an embodiment of the present disclosure.

[0068] like Figure 4B As shown, in some embodiments, adding noise to the first image multiple times to obtain the second image can be further implemented by the following steps.

[0069] First, a first noise ε is added to the first image 4084. 1 , and obtain a second intermediate image 4122. The second intermediate image 4122 is obtained according to the first fusion parameter α 1 The first image 4084 and the first noise ε 1 Optionally, the weight corresponding to the first image 4084 may be a first fusion parameter α 1 The square root of the first noise ε 1 The corresponding weight can be 1 minus the first fusion parameter α 1The square root of the difference between the first image 4084 and the first noise ε 1 The sum of the squares of the weights is 1), the second intermediate image 4122 is the first image 4084 and the first noise ε 1 The weighted sum of .

[0070] Then, if Figure 4B As shown, a second noise ε may be further added to the second intermediate image 4122. 2 , and obtain a third intermediate image 4124. The third intermediate image 4124 is obtained according to the second fusion parameter α 2 The second intermediate image 4122 and the second noise ε 2 Optionally, the weight corresponding to the second intermediate image 4122 may be the second fusion parameter α 2 The square root of the second noise ε 2 The corresponding weight can be 1 minus the second fusion parameter α 2 The square root of the difference between the second intermediate image 4122 and the second noise ε 2 The sum of the squares of the weights is 1), the third intermediate image 4124 is the sum of the second intermediate image 4122 and the second noise ε 2 The weighted sum of .

[0071] Among them, the second fusion parameter α 2 is smaller than the first fusion parameter α 1 As mentioned above, the first noise ε 1 The corresponding weight can be 1 minus the first fusion parameter α 1 The square root of the difference, the second noise ε 2 The corresponding weight can be 1 minus the second fusion parameter α 2 The square root of the difference, in other words, when the second fusion parameter α 2 is smaller than the first fusion parameter α 1 When the second noise ε 2 The corresponding weight may be greater than the first noise ε 1 That is to say, in this embodiment, as the number of times noise is added increases, the weight of the noise also gradually increases. In this way, on the one hand, a small noise can be added in the initial stage of adding noise to better preserve the details of the original image, and on the other hand, after the noise is added, the second image can be closer to a noise image, so that the third model 406 can better generate a high-resolution image.

[0072] According to the above logic, a third noise ε may be further added to the third intermediate image 4124. 3, a fourth intermediate image (not shown in the figure) is obtained, wherein the fourth intermediate image is obtained according to the third fusion parameter α 3 The third intermediate image 4124 and the third noise ε 3 Optionally, the weight corresponding to the third intermediate image 4124 may be a third fusion parameter α 3 The square root of the third noise ε 3 The corresponding weight can be 1 minus the third fusion parameter α 3 The square root of the difference between the third intermediate image 4124 and the third noise ε 3 The sum of the squares of the weights is 1), the fourth intermediate image is the third intermediate image 4124 and the third noise ε 3 The weighted sum of . Among them, the third fusion parameter α 3 is smaller than the second fusion parameter α 2 .

[0073] The above iterative process is repeated until t noise additions are completed to obtain the second image 4126.

[0074] In step 208, a target image is generated according to the second image.

[0075] Alternatively, if Figure 4A As shown, the second image (the first image 4084 with noise added) can be input into the third model 406 to generate a high-resolution target image.

[0076] In some embodiments, in order to obtain a better super-resolution image, the third model 406 may be a control network model (ControlNet), so that the effect of the output image can be controlled by adding conditional constraints.

[0077] In some embodiments, Figure 2B As shown, generating a target image according to the second image includes:

[0078] In step 2082, color constraints 4104 and / or text constraints 4106 are obtained based on the second image.

[0079] In order to impose sufficiently strong constraints on the content generated by ControlNet and avoid excessive deviation between the generated image content and the target area selected by the user, in this embodiment, color constraints 4104 and / or text constraints 4106 are selected to constrain the output results.

[0080] In some embodiments, obtaining the color constraint based on the second image includes: downsampling the second image to a target size (e.g., 16×16) as the color constraint 4104. In this way, the color constraint 4104 obtained by downsampling can well reflect the characteristics of the target area 302, thereby ensuring that the generated image content is consistent with the content of the target area 302 selected by the user.

[0081] In some embodiments, obtaining the text constraint based on the second image includes: generating text corresponding to the second image as the text constraint 4106 according to the second image. In this way, by converting image information into text information as the text constraint, a constraint condition is provided from another perspective, and it can also be ensured that the generated image content is consistent with the content of the target area 302 selected by the user.

[0082] Optionally, based on the second image, text corresponding to the second image is generated as the text constraint, including: inputting the second image into a pre-trained text generation model (for example, a Bootstrapping Language-Image Pre-training (BLIP) model), and outputting the text corresponding to the second image.

[0083] The text generation model can generate text reflecting the content of the image based on the image. Using the text as a text constraint can well reflect the content of the second image, thereby ensuring that the generated image content is consistent with the content of the target area 302 selected by the user.

[0084] In step 2084 , the target image is generated according to the second image, the color constraint 4104 and the text constraint 4106 .

[0085] Optionally, the target image is generated according to the second image, the color constraint and the text constraint, including: inputting the second image, the color constraint 4104 and the text constraint 4106 into a pre-trained control network model (ControlNet), and outputting the target image, wherein the color constraint 4104 and the text constraint 4106 can be used as control conditions of ControlNet to constrain the output result.

[0086] In this way, the target image 4086 is output by inputting the color constraint 4104, the text constraint 4106 and the second image into ControlNet at the same time, so that the target image 4086 can better reflect the details of the original image.

[0087] Figure 3B The embodiment of the present disclosure is shown Figure 3AFIG. 3 is a schematic diagram of an exemplary image 310 after a zoomed-in target area 302 is shown.

[0088] like Figure 3B As shown, the image 310 processed by the method 200 of the embodiment of the present disclosure can be a very clear high-resolution image compared to the original target area 302 .

[0089] Figure 3C Shows the Figure 3A FIG. 3 is a schematic diagram of an exemplary image 320 after the target area 302 is enlarged.

[0090] like Figure 3C As shown, the image 320 obtained after the target area 302 is enlarged in a conventional manner is a blurred image.

[0091] By comparison Figure 3B The image 310 and Figure 3C It can be seen from the image 320 that by using the method 200 of the embodiment of the present disclosure, a high-definition image consistent with the content of the target area 302 can be obtained, thereby achieving an infinite magnification effect.

[0092] It can be seen from the above embodiments that the image processing method provided by the embodiments of the present disclosure can realize infinite magnification of any area in the image (referring to the user arbitrarily specifying an area in the image and achieving clarity). The magnification effect of this method has a powerful and amazing effect on the basis of ensuring the consistency between the super-resolution result and the original image.

[0093] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0094] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] The embodiment of the present disclosure also provides a computer device for implementing the above method 200. Figure 5FIG. 5 is a schematic diagram showing the hardware structure of an exemplary computer device 500 provided in an embodiment of the present disclosure. The computer device 500 can be used to implement Figure 1 The server 106 can also be used to implement Figure 1 In some scenarios, the computer device 500 can also be used to implement Figure 1 The database server 108.

[0096] like Figure 5 As shown, the computer device 500 may include: a processor 502, a memory 504, a network module 506, a peripheral interface 508 and a bus 510. The processor 502, the memory 504, the network module 506 and the peripheral interface 508 are connected to each other in communication within the computer device 500 through the bus 510.

[0097] Processor 502 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 502 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 502 may also include multiple processors integrated into a single logical component. For example, Figure 5 As shown, processor 502 may include a plurality of processors 502a, 502b, and 502c.

[0098] The memory 504 may be configured to store data (eg, instructions, computer code, etc.). Figure 5 As shown, the data stored in the memory 504 may include program instructions (e.g., program instructions for implementing the method 200 of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 502 may also access the program instructions and data stored in the memory 504, and execute the program instructions to operate on the data to be processed. The memory 504 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 504 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0099] The network interface 506 can be configured to provide the computer device 500 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the above specific examples.

[0100] The peripheral interface 508 can be configured to connect the computer device 500 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0101] The bus 510 may be configured to transmit information between various components of the computer device 500 (e.g., the processor 502, the memory 504, the network interface 506, and the peripheral interface 508), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0102] It should be noted that, although the architecture of the above-mentioned computer device 500 only shows the processor 502, the memory 504, the network interface 506, the peripheral interface 508 and the bus 510, in the specific implementation process, the architecture of the computer device 500 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the above-mentioned computer device 500 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0103] The embodiment of the present disclosure also provides an image processing device. Figure 6 FIG. 6 is a schematic diagram of an exemplary device 600 provided by an embodiment of the present disclosure. Figure 6 As shown, the device 600 can be used to implement the method 200 and can further include the following modules.

[0104] The determination module 602 is configured to: determine a target area of ​​the image;

[0105] The magnification module 604 is configured to: magnify the target area at least twice to obtain a first image;

[0106] The noise module 606 is configured to: perform noise processing on the first image to obtain a second image;

[0107] The generating module 608 is configured to generate a target image according to the second image.

[0108] In some embodiments, the amplification module 604 is configured to:

[0109] Using a pre-trained first diffusion model, the target area is magnified by multiples to obtain a first intermediate image;

[0110] The first intermediate image is magnified by multiples using a pre-trained second diffusion model to obtain the first image.

[0111] In some embodiments, the noise module 606 is configured to add noise to the first image multiple times to obtain the second image.

[0112] In some embodiments, the noise module 606 is configured to:

[0113] Adding first noise to the first image to obtain a second intermediate image; wherein the second intermediate image is obtained by weighting the first image and the first noise according to a first fusion parameter;

[0114] Adding second noise to the second intermediate image to obtain a third intermediate image; wherein the third intermediate image is obtained by weighting the second intermediate image and the second noise according to a second fusion parameter, and the second fusion parameter is smaller than the first fusion parameter.

[0115] In some embodiments, the generating module 608 is configured to:

[0116] Acquire a color constraint and a text constraint based on the second image;

[0117] The target image is generated according to the second image, the color constraint and the text constraint.

[0118] In some embodiments, the generating module 608 is configured to: downsample the second image to a target size as the color constraint.

[0119] In some embodiments, the generating module 608 is configured to: generate text corresponding to the second image according to the second image as the text constraint.

[0120] In some embodiments, the generation module 608 is configured to: input the second image into a pre-trained text generation model, and output text corresponding to the second image.

[0121] In some embodiments, the generation module 608 is configured to: input the second image, the color constraint and the text constraint into a pre-trained control network model, and output the target image.

[0122] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0123] The device of the above embodiment is used to implement the corresponding method 200 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0124] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute method 200 described in any of the above embodiments.

[0125] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0126] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method 200 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] Based on the same inventive concept, corresponding to the method 200 in any of the above embodiments, the present disclosure further provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors so that the processors execute the method 200. Corresponding to the execution subject corresponding to each step in each embodiment of the method 200, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0128] The computer program product of the above embodiment is used to enable the processor to execute the method 200 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0129] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0130] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0131] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0132] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, comprising: Determine the target area of ​​the image; In response to a zoom-in instruction for the target area, the target area is zoomed in at least twice to obtain a first image; Performing noise processing on the first image to obtain a second image; A target image is generated according to the second image.

2. The method of claim 1, wherein: The target area is magnified at least twice to obtain a first image, comprising: Using a pre-trained first diffusion model, the target area is magnified by multiples to obtain a first intermediate image; The first intermediate image is magnified by multiples using a pre-trained second diffusion model to obtain the first image.

3. The method of claim 1, wherein: Performing noise processing on the first image to obtain a second image includes: Noise is added to the first image multiple times to obtain the second image.

4. The method of claim 3, wherein: Adding noise to the first image multiple times to obtain the second image includes: Adding first noise to the first image to obtain a second intermediate image; wherein the second intermediate image is obtained by weighting the first image and the first noise according to a first fusion parameter; Adding second noise to the second intermediate image to obtain a third intermediate image; wherein the third intermediate image is obtained by weighting the second intermediate image and the second noise according to a second fusion parameter, and the second fusion parameter is smaller than the first fusion parameter.

5. The method of claim 1, wherein: Generating a target image according to the second image includes: Acquire a color constraint and a text constraint based on the second image; The target image is generated according to the second image, the color constraint and the text constraint.

6. The method of claim 5, wherein: Acquiring a color constraint and a text constraint based on the second image includes: downsampling the second image to a target size as the color constraint; According to the second image, text corresponding to the second image is generated as the text constraint.

7. The method of claim 5, wherein: Generating the target image according to the second image, the color constraint and the text constraint includes: The second image, the color constraint and the text constraint are input into a pre-trained control network model, and the target image is output.

8. An image processing device, comprising: The determination module is configured to: determine a target area of ​​the image; an enlargement module, configured to: in response to an enlargement instruction for the target area, enlarge the target area at least twice to obtain a first image; A noise module is configured to: perform noise processing on the first image to obtain a second image; The generating module is configured to generate a target image according to the second image.

9. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 7.

Images