Image processing methods and apparatus

Abstract

This application discloses an image processing method and apparatus, belonging to the field of electronic technology. The method includes: performing image segmentation processing based on a first image to obtain a first subject image; performing a first image enhancement processing on the first subject image to obtain a second subject image; and performing a second image enhancement processing on the first image based on the first image and its description information to obtain a second image; and performing image fusion processing based on the second subject image and the second image to obtain a third image, wherein the third image includes a background image from the second subject image and the second image.

Description

Technical Field

[0001] This application belongs to the field of electronic technology, specifically relating to an image processing method and apparatus. Background Technology

[0002] Artificial Intelligence Generated Content (AIGC) technology has been widely applied in various fields with significant market value, including literary creation, assisted programming, image processing, and video generation. In image processing, in addition to the content of the image itself, users have very high requirements for the image quality and resolution of AIGC-generated images. For example, users may require 4K, 8K, or even higher resolution images.

[0003] In related technologies, in order to balance the cost of image processing and image quality, low-resolution image processing is usually adopted, and then efficient super-resolution algorithms are used to further improve the image quality of the low-resolution image to obtain an image with a resolution that meets the user's needs.

[0004] However, in the above methods, the process of using super-resolution algorithms to enhance image quality may result in image detail artifacts, such as meshing and blurring, leading to poor image quality. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method and apparatus that can improve the image quality of the processed image.

[0006] In a first aspect, embodiments of this application provide an image processing method, which includes: performing image segmentation processing on a first image to obtain a first subject image; performing first image enhancement processing on the first subject image to obtain a second subject image; and performing second image enhancement processing on the first image based on the first image and its description information to obtain a second image; and performing image fusion processing on the second subject image and the second image to obtain a third image, wherein the third image includes a background image from the second subject image and the second image.

[0007] Secondly, embodiments of this application provide an image processing apparatus, which includes: a processing module; the processing module is configured to perform image segmentation processing based on a first image to obtain a first subject image; perform a first image enhancement processing on the first subject image to obtain a second subject image; perform a second image enhancement processing on the first image based on the first image and its description information to obtain a second image; and perform image fusion processing based on the second subject image and the second image to obtain a third image, the third image including a background image from the second subject image and the second image.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program / program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, the electronic device can segment a first image to obtain a first subject image. Then, the electronic device can perform a first image enhancement process on the first subject image, and based on the first image and its descriptive information, perform a second image enhancement process. That is, the electronic device can perform different image enhancements on the subject image and the original image, providing a more suitable image enhancement for the subject image. Simultaneously, since the descriptive information of the first image is used during the enhancement process, this information can be used to improve the ability to restore image details during the image enhancement process. Therefore, the electronic device can perform image fusion using a second subject image with better image enhancement and a second image with better detail restoration to obtain a processed third image. In this way, the consistency of the subject and the restoration of background details can be balanced, thereby improving the image quality of the processed image. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts of an image processing method provided in the embodiments of this application;

[0014] Figure 2A This is one of the schematic diagrams illustrating an image provided in an embodiment of this application;

[0015] Figure 2B This is a second example of an image provided in an embodiment of this application;

[0016] Figure 2C This is the third example of an image provided in the embodiments of this application;

[0017] Figure 3 This is a second flowchart of an image processing method provided in an embodiment of this application;

[0018] Figure 4 This is the third flowchart of an image processing method provided in the embodiments of this application;

[0019] Figure 5 This is the fourth flowchart of an image processing method provided in the embodiments of this application;

[0020] Figure 6 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0021] Figure 7 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0022] Figure 8 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects. For example, a first object can be one or more, where "more" means at least two. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] The terms "at least one" and "at least one of" in this application's specification refer to any one, any two, or a combination of two or more of the included objects. For example, "at least one of a, b, and c" can mean "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple, and multiple means at least two. Similarly, "at least two" means two or more, and its meaning is similar to "at least one". The identifiers in this application are text, symbols, images, etc., used to indicate information, and can use controls or other containers as carriers for displaying information, including but not limited to text identifiers, symbol identifiers, and image identifiers.

[0026] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0027] AIGC refers to the automatic generation of various types of content through artificial intelligence technology, including but not limited to text, images, audio, and video. The core of AIGC lies in using technologies such as deep learning, natural language processing (NLP), and computer vision to enable machines to understand, generate, and optimize content, thereby achieving automation and intelligence in the content production process.

[0028] A 1K image typically refers to an image with approximately 1000 pixels in the horizontal direction, such as 1024. A common standard is 1280×720 pixels, or 720p.

[0029] 4K refers to an image with approximately 4000 pixels horizontally, such as 3840 or 4096. Common standards are: Consumer-grade 4K: 3840×2160 pixels, widely used in televisions and monitors. Cinema-grade 4K: 4096×2160 pixels, used for digital cinema projection.

[0030] Image super-resolution (ISR) is a technique that uses algorithms or models to recover high-resolution (HR) images from low-resolution (LR) images. It aims to improve the detail clarity and spatial resolution of images and is widely used in fields such as medical imaging, satellite remote sensing, security monitoring, and digital entertainment.

[0031] The Structural Similarity Index (SSIM) is a metric used to measure the similarity between two images, such as the similarity between the original image and a compressed or processed image. SSIM focuses on the structural information of the image, comprehensively evaluating image quality from three dimensions: brightness, contrast, and structure. Compared to traditional metrics such as Peak Signal-to-Noise Ratio (PSNR), it is more consistent with the perceptual characteristics of the human visual system (HVS).

[0032] Specifically, for high SSIM, the closer it is to 1, the higher the similarity between the model-generated image and the original image in terms of structure, brightness, and contrast, and the closer the visual quality is to the real scene. For example, in super-resolution reconstruction tasks, high SSIM means that the reconstructed image retains complete details and has clear edges. Conversely, for low SSIM, the closer it is to 0, the more severe the distortion of the image structure, and the more likely there will be blurring, artifacts, or color deviation. For example, if compression algorithms cause a significant decrease in SSIM, it will reduce the user's viewing experience.

[0033] Today, AIGC technology is widely used in various fields with high market value, such as literary creation, assisted programming, image processing, and video generation. In the field of image processing, in addition to the content of the image itself, users have very high requirements for the image quality and resolution of AIGC-generated images. Traditional 1K resolution images can no longer meet user needs. 4K, 8K, or higher resolution images are needed to better satisfy users' visual experience.

[0034] Image processing models require significant computational power, and directly generating high-quality, high-resolution images from these models incurs substantial computational costs and time consumption. Furthermore, directly upscaling a 1K image generated by the model to 4K resolution results in image detail artifacts, such as meshing and blurring. Therefore, related technologies commonly employ low-resolution image processing followed by efficient super-resolution algorithms to further enhance the image quality of the low-resolution image.

[0035] However, for super-resolution algorithms using related technologies, the goals of maintaining subject consistency and background detail diversity are contradictory. Therefore, problems often arise where enhanced subject consistency is not maintained, or enhanced background detail is insufficient. Specifically, if super-resolution algorithms over-control subject consistency during training, it leads to insufficient enhancement of overall image detail; conversely, if they overemphasize background detail diversity, it results in distortion of facial features. Thus, finding a balance between these two aspects is a pressing issue that needs to be addressed.

[0036] Furthermore, while portrait enhancement models in related technologies can guarantee a certain degree of consistency in the representation of a person—that is, the overall outline and coarse-grained information of facial features remain unchanged—there is still room for optimization in controlling fine-grained facial details. For example, super-resolution algorithms can easily create non-existent moles on facial skin or exaggerate wrinkles on the enhanced face. These subtle changes on the face are easily perceived by users, thus affecting the overall user experience.

[0037] This will result in poor image quality after the image is enhanced using super-resolution algorithms in related technologies.

[0038] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0039] The image processing method provided in this application can be applied to scenarios involving image quality enhancement processing based on AIGC-generated images. For example, it can convert a 1K image into a 4K image using an image super-resolution algorithm.

[0040] The image processing method provided in this application embodiment will be illustrated below using a specific scenario as an example.

[0041] When a user wants to adjust a lower-resolution image to a higher-resolution image, such as adjusting a 1K image to a 4K image, the user can use an electronic device to perform image segmentation on the original image to obtain the main subject image contained in the original image. Then, the electronic device can enhance this main subject image to obtain an enhanced main subject image, and based on the original image and its corresponding descriptive information, perform image enhancement on the original image to obtain an enhanced original image, such as a preliminary enhanced image. The electronic device can then perform a fusion process based on the enhanced main subject image and the preliminary enhanced image to obtain the final enhanced image. This final enhanced image can include both the enhanced main subject image and the background image from the preliminary enhanced image. That is, the final enhanced image can simultaneously include the main subject image with better image enhancement and the background image with better detail restoration, thus balancing the consistency of the subject and the restoration of background details, improving the image quality of the final generated third image.

[0042] It should be noted that the above scenarios are merely illustrative examples of possible applications of the embodiments of this application. In actual implementation, the embodiments of this application can also be applied to any possible scenarios of image processing, such as image enhancement processing, and are not limited here.

[0043] Based on the scenarios described in the embodiments of this application, the image processing method provided in this application allows an electronic device to segment a first image to obtain a first subject image within the first image. Then, the electronic device can perform a first image enhancement process on the first subject image, and based on the first image and its descriptive information, perform a second image enhancement process. That is, the electronic device can perform different image enhancements on the subject image and the original image, providing a more suitable image enhancement for the subject image. Simultaneously, since the descriptive information of the first image is used during the enhancement process, this information can improve the ability to restore image details during the image enhancement process. Therefore, the electronic device can perform image fusion using a second subject image with better image enhancement and a second image with better detail restoration to obtain a processed third image. This approach balances the consistency of the subject and the restoration of background details, thereby improving the image quality of the processed image.

[0044] The image processing method provided in this application is executed by an image processing device, which can be an electronic device, or a functional module or entity within an electronic device. This application does not limit the specific implementation of this method. The following will use an electronic device as an example to illustrate the image processing method provided in this application.

[0045] This application provides an image processing method. Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application is shown. Figure 1 As shown, the image processing method provided in this application embodiment may include the following steps 201 to 203.

[0046] Step 201: The electronic device performs image segmentation processing based on the first image to obtain the first main image.

[0047] In some embodiments of this application, the first image described above can be a low-resolution image. For example, the first image can be a 1K image, that is, the resolution of the first image can be 1280×720.

[0048] In some embodiments of this application, the first subject image can be the image region corresponding to the subject in the first image.

[0049] In some embodiments of this application, the shape of the first main image can be a regular geometric shape or an irregular shape. The specific shape can be determined according to actual needs, and this application does not impose any limitations on it.

[0050] In some embodiments of this application, the aforementioned subject can be the main object represented in the first image, that is, it can be the visual center and focus of the first image.

[0051] In some embodiments of this application, the aforementioned subject may be a person.

[0052] In some embodiments of this application, when a first image contains multiple objects, the electronic device can automatically determine the subject among the multiple objects, or the electronic device can determine the subject among the multiple objects based on user input.

[0053] In some embodiments of this application, the information referenced by the electronic device to automatically determine the subject among multiple objects may include, but is not limited to, at least one of the following: the position of the object in the image, the ratio of the object size to the image size, the distance of the object from the center point of the image, the distance of the object from the focus point of the image, and the sharpness of the object.

[0054] For example, suppose the first image described above is as follows: Figure 2A Image A shows a boy kneeling in front of a building, holding a beverage cup. The electronic device can first perform object detection on image A to determine the specific objects it contains, such as two objects: a person (the boy) and a building. Then, the electronic device can automatically identify the person as the main object; that is, the first main image obtained by the electronic device can be the image corresponding to the person (the boy).

[0055] In some embodiments of this application, the electronic device can input the first image described above into a portrait detection and instance segmentation model to obtain the specific pixel coordinates of the portrait and the background.

[0056] In some embodiments of this application, the above-mentioned human face detection and instance segmentation model can be a model built using SegFormer as a framework, and the mean intersection over union (mIoU) index of the model can be approximately 90%.

[0057] In some embodiments of this application, the electronic device can obtain the binary mask image corresponding to the first image through the above-described portrait detection and instance segmentation model, and further obtain the first subject image based on the binary mask image.

[0058] It should be noted that the aforementioned binary mask image can be an image with the exact same size as the original image, and each pixel value in the binary mask image can be used to represent category information. For example, in the binary mask image, pixels with a value of 255 or 1, i.e., white areas, can be used to represent image areas where "operation is allowed," and these parts of the image can be adjusted accordingly during image processing; pixels with a value of 0, i.e., black areas, can be used to represent "operation is prohibited," and these parts of the image will remain unchanged and will not be affected by subsequent steps.

[0059] In some embodiments of this application, when the electronic device acquires the Mask image corresponding to the first image, the electronic device can multiply the Mask image with the first image to achieve image segmentation by extracting or retaining the image corresponding to the white area in the Mask image, thereby obtaining the aforementioned first main image.

[0060] For example, suppose the subject in image A is a person. An electronic device can input image A into a human detection and instance segmentation model to obtain, as shown below. Figure 2B The Mask image 1 shown indicates that the person in image A is identified as the image region to be segmented subsequently. Simultaneously, the electronic device can acquire images such as... Figure 2C The Mask image 2 shown is the image region in Image A that is defined as the background excluding the person, which needs to be segmented later.

[0061] Step 202: The electronic device performs a first image enhancement process on the first subject image to obtain a second subject image, and performs a second image enhancement process on the first image based on the first image and its description information to obtain the second image.

[0062] It is understandable that the electronic device can provide different image enhancement processes for the first subject image and the first image to obtain a subject image with better enhancement effect, thus satisfying the consistency of the subject; and the electronic device can acquire a second image including a background with better detail restoration effect, so that the subsequently generated third image can take into account both the consistency of the subject and the restoration of background details.

[0063] In some embodiments of this application, the different image enhancement processes described above may include, but are not limited to, at least one of the following: using different image enhancement models, using the same image enhancement model but with different model parameters, or using the same image enhancement model but focusing on different objects. The specific details can be determined according to actual needs, and this application does not limit the specific implementation.

[0064] In some embodiments of this application, combined with Figure 1 ,like Figure 3 As shown. After step 201 above, the image processing method provided in this application embodiment may further include step 301 below, and step 202 above can be specifically implemented by steps 202a and 202b below.

[0065] Step 301: The electronic device performs upsampling processing on the first image and the first subject image respectively to obtain the fourth image and the third subject image.

[0066] In some embodiments of this application, the resolution of the fourth image may be greater than that of the first image.

[0067] In some embodiments of this application, the resolution of the third subject image is greater than that of the first subject image.

[0068] In some embodiments of this application, the above upsampling process may include, but is not limited to, at least one of the following methods: interpolation, transposed convolution, subpixel convolution, and dilated convolution.

[0069] It is understandable that electronic devices can obtain a corresponding high-resolution image from a low-resolution image through upsampling, and then perform corresponding image enhancement processing on the high-resolution image to improve the display effect of the high-resolution image, thus achieving an improvement in image quality.

[0070] In some embodiments of this application, the resolution of the image after the upsampling process can be greater than or equal to a preset resolution. This preset resolution can be a default setting by the electronic device or a resolution manually set by the user, representing the desired resolution of the final generated image.

[0071] For example, if a user wants to obtain a 4K resolution image based on a 1K resolution image, the electronic device can upsample the first image and the first subject image from 1K resolution to 4K resolution, thus obtaining a 4K resolution fourth image and a third subject image, for subsequent image enhancement processing. Alternatively, the electronic device can upsample the first image and the first subject image from 1K resolution to a resolution higher than 4K, such as 8K, for subsequent image enhancement processing. Then, after obtaining the final 8K resolution image, it can downsample the 8K resolution image to the user's desired 4K resolution to obtain a 4K image with better display quality. This is the process of image enhancement processing achieved through the principle of supersampling technology.

[0072] It should be noted that the specific implementation method of "downsampling the 8K resolution image to the 4K resolution required by the user" can be found in the relevant technologies for the implementation method of image downsampling, and will not be repeated here in the embodiments of this application.

[0073] It should be noted that the higher the image resolution, the better the overall display effect of the image.

[0074] Understandably, since 8K resolution images have a higher resolution than 4K images, the image enhancement processing performed when an electronic device upsamples an image to 8K resolution is essentially done on a finer pixel grid. This results in smoother image edges, cleaner noise, and more natural color transitions. Then, downsampling the enhanced 8K image to 4K resolution is equivalent to an anti-aliasing filter, removing jagged edges and artifacts that may have occurred during image processing. This results in a final 4K resolution image with cleaner edges, fewer artifacts, and more refined image quality.

[0075] It should be noted that the electronic device may first perform upsampling processing on the first image to obtain a first image with high resolution, namely the fourth image mentioned above, and then perform image segmentation processing on the fourth image similar to step 201 above to obtain the main image, such as the third main image mentioned above. That is to say, the order of image segmentation processing similar to step 201 and upsampling processing similar to step 301 on the original image is not limited in this embodiment of the application. The above 3 is an example of performing image segmentation processing on the original image first and then performing upsampling processing.

[0076] Step 202a: The electronic device performs first image enhancement processing on the third subject image through the first image enhancement model to obtain the second subject image.

[0077] In some embodiments of this application, the first image enhancement model described above can be a model constructed based on a portrait super-resolution algorithm. That is, the first...

[0078] In some embodiments of this application, the first image enhancement model described above can be a model with high SSIM (Simplified Simulation), so that the second subject image obtained by the electronic device can maintain high consistency with the original image while enriching portrait details. The original image can be either the first subject image or a third subject image.

[0079] For example, the SSIM of the first image enhancement model can be greater than 0.9, that is, the second subject image processed by the first image enhancement model can be highly similar to the original reference in terms of structure, brightness and contrast, and can make local details such as edges and textures and global structures such as the shape of the subject object almost undistorted.

[0080] Step 202b: The electronic device uses the second image enhancement model to perform second image enhancement processing on the fourth image based on the description information of the fourth image and the first image, to obtain the second image.

[0081] In some embodiments of this application, the second image enhancement model described above can be used to enhance the image details of the background image in the fourth image described above.

[0082] In some embodiments of this application, the structural similarity index of the second image enhancement model may be less than the structural similarity index of the first image enhancement model.

[0083] For example, the SSIM of the second image enhancement model can be greater than 0.5 and less than 0.7. That is, compared with the first image model, the second image enhancement model has a stronger tendency for diversity, thereby obtaining a high-resolution image with richer background details.

[0084] Understandably, the first image enhancement model focuses on preserving the original structure of the image, that is, the accurate expression of the structure, such as maintaining a high consistency between local details such as the skin texture of a person and the original image; the second image enhancement model focuses on the stylistic expression of the image.

[0085] It should be noted that the description information of the first image mentioned above can be found in the relevant descriptions of steps 601 and 602 below, which will not be repeated here in the embodiments of this application.

[0086] In this embodiment, the electronic device can upsample the first image, first enlarging it from a 1K image to a 4K image. This reduces the computational cost and time required for the model to directly generate high-quality and high-resolution images from the 1K image. Simultaneously, the electronic device can perform highly controllable detail enhancement on the image segmented portrait based on two different image enhancement models, and perform more diverse detail enhancement on the background content. This balances the consistency of the subject and the restoration of background details, improving the image quality of the final generated third image.

[0087] In some embodiments of this application, before “performing a first image enhancement process on the first subject image to obtain a second subject image” in step 202 above, the image processing method provided in the embodiments of this application may further include the following step 501.

[0088] Step 501: The electronic device performs image smoothing processing on the first subject image through a bilateral filter to obtain the smoothed first subject image.

[0089] Understandably, taking a person as the subject as an example, for super-resolution algorithms of related technologies, when the image contains a person, although the general facial features and outline of the portrait can be preserved after enhancement processing, fine-grained information such as skin texture is prone to over-generating. Therefore, a bilateral filter can be used to preprocess the portrait portion of the input image to ensure that the enhanced image does not excessively enhance the original noise, thereby avoiding facial artifacts.

[0090] In some embodiments of this application, the parameters of the bilateral filter may include: filter window diameter, color space standard deviation, and spatial standard deviation.

[0091] For example, the filter window diameter can be set to 4-14, the color space standard deviation can be set to 30-40, and the space standard deviation can be set to 20-30. It should be noted that the specific parameter settings can be determined according to actual needs. This embodiment is merely an example and does not limit the specific parameter settings.

[0092] For example, the filter window diameter can be 9, the standard deviation of the color space can be 35, and the standard deviation of the space can be 25.

[0093] It should be noted that, for the specific implementation method of how to perform image smoothing processing on the first subject image through a bilateral filter to obtain the smoothed first subject image, please refer to the usage method of bilateral filters in related technologies, and will not be described in detail in the embodiments of this application.

[0094] In this embodiment, the electronic device can preprocess the main body portion of the first image, i.e., the first main body image, using a bilateral filter to make the original noise of the image relatively smooth. This avoids excessive enhancement of the original noise in the subsequent image enhancement stage of the main body image, thereby effectively reducing or avoiding artifacts in the second main body portrait. In this way, the consistency of the main body can be guaranteed, and the image quality of the finally generated third image can be improved.

[0095] In some embodiments of this application, before "based on the first image and the description information of the first image, perform a second image enhancement process on the first image to obtain a second image" in step 202 above, the image processing method provided in the embodiments of this application may also include the following steps 601 and 602.

[0096] Step 601: The electronic device obtains the feature information of the first image based on the first image through the graph-to-text model.

[0097] Step 602: The electronic device generates descriptive information of the first image based on the feature information of the first image.

[0098] In some embodiments of this application, the above-mentioned graph-to-text model can be the Florence-2 model, or a model built on the framework of the Florence-2 model, or any other feasible graph-to-text model.

[0099] In some embodiments of this application, the feature information of the first image may include, but is not limited to, at least one of the following: feature information of the object in the first image, and feature information of the background in the first image.

[0100] In some embodiments of this application, the above descriptive information can be used to describe the main object and background in the first image.

[0101] For example, when the first image includes a person, the descriptive information corresponding to the person may include, but is not limited to, at least one of the following: person's gender, person's clothing, person's location, person's posture and action, person's appearance, and person's age;

[0102] When the first image includes an animal, the descriptive information corresponding to the animal may include, but is not limited to, at least one of the following: animal species, animal color, human position, animal posture and movement, and number of animals;

[0103] If the first image includes a plant, the descriptive information corresponding to the plant may include, but is not limited to, at least one of the following: plant variety, plant color, plant location, and plant morphology;

[0104] If the first image includes a building, the descriptive information corresponding to the building may include, but is not limited to, at least one of the following: architectural style and architectural color.

[0105] In some embodiments of this application, the descriptive information corresponding to the background described above may include, but is not limited to, at least one of the following: weather conditions and environmental scene.

[0106] In some embodiments of this application, the above description information may also include information for describing image parameters.

[0107] In some embodiments of this application, the above image parameters may include, but are not limited to, at least one of the following: hue, brightness, saturation, contrast, color temperature, and sharpness.

[0108] It should be noted that, for the specific implementation method of how to generate descriptive information of the first image based on the first image through the graph-to-text model, please refer to the usage method of the graph-to-text model in related technologies, and the embodiments of this application will not be described in detail here.

[0109] In this embodiment of the application, the electronic device can obtain the description information of the first image through the first image, that is, it can obtain the image content contained in the original image. Therefore, in the subsequent process of enhancing the first image as a whole, the description information can be used to ensure the diversity of background details, thereby improving the image quality of the finally generated third image.

[0110] Step 203: The electronic device performs image fusion processing based on the second subject image and the second image to obtain the third image.

[0111] In some embodiments of this application, the third image described above may include the second subject image and the background image in the second image.

[0112] In some embodiments of this application, the electronic device can determine the display position of the main image in the second image based on the second main image and the mask image corresponding to the main image, and then perform image fusion processing based on the second main image and the second image in combination with the display position to obtain a third image.

[0113] Understandably, electronic devices can perform image fusion of the enhanced portrait and background based on the pixel positions obtained from the segmentation model.

[0114] In some embodiments of this application, combined with Figure 1 ,like Figure 4 As shown, step 203 can be implemented through steps 203a and 203b.

[0115] Step 203a: The electronic device updates the main image in the second image to the second main image, thus obtaining the fifth image.

[0116] In some embodiments of this application, the above-described updating method may include, but is not limited to, any of the following:

[0117] The main image in the second image is directly overlaid with the second main image;

[0118] The main image in the second image is cut out, and then the main image is filled into the background image corresponding to the second image.

[0119] Step 203b: The electronic device performs color correction processing on the fifth image to obtain the third image.

[0120] In some embodiments of this application, an electronic device can extract low-frequency information from the original image and high-frequency information from the fused image using wavelet transform, and then sum the two to ensure that the enhanced image maintains the same color tone as the original image while preserving the detail information of the fused image. This enables color correction processing of the fused image.

[0121] It is understandable that electronic devices can extract low-frequency information from the first image and high-frequency information from the fifth image through wavelet transform in order to perform color correction processing on the fifth image and obtain the corrected third image.

[0122] In some embodiments of this application, the above-mentioned color correction processing may include, but is not limited to, at least one of the following: brightness correction processing, contrast correction processing, highlight correction processing, color temperature correction processing, saturation correction processing, and edge blending processing.

[0123] In this embodiment, the electronic device can update the main image in the second image to a second main image and perform color correction on the updated image to reduce or avoid the visual difference between the main object and the background in the final generated third image, thereby enhancing the visual realism of the third image and improving image quality.

[0124] In the image processing method provided in this application embodiment, the electronic device can segment a first image to obtain a first subject image in the first image. Then, the electronic device can perform a first image enhancement process on the first subject image, and based on the first image and its descriptive information, perform a second image enhancement process on the first image. That is, the electronic device can perform different image enhancements on the subject image and the original image, thus providing a more suitable image enhancement for the subject image. Simultaneously, since the descriptive information of the first image is used during the enhancement process, this descriptive information can be used to improve the ability to restore image details during the image enhancement process. Therefore, the electronic device can perform image fusion using a second subject image with better image enhancement and a second image with better detail restoration to obtain a processed third image. In this way, the consistency of the subject and the restoration of background details can be balanced, thereby improving the image quality of the processed image.

[0125] This application provides an image processing method. Figure 5 A flowchart of an image processing method provided in an embodiment of this application is shown. The following will illustrate this method as follows: Figure 2A The image processing method provided in this application embodiment is illustrated using image A, which shows a boy as the main subject.

[0126] The model used in this application embodiment can be a model built based on a super-resolution algorithm framework. Furthermore, it can include three modules, such as: a portrait background perception module, a background detail enhancement module, and a portrait detail enhancement module.

[0127] Among them, the aforementioned portrait background perception module can distinguish the foreground portrait from the background through an instance segmentation algorithm, providing prior information for the subsequent enhancement module.

[0128] The aforementioned background detail enhancement module can provide more diverse and detailed enhancements to the distinguished background content.

[0129] The aforementioned portrait detail enhancement module can perform highly controllable detail restoration on the segmented portrait.

[0130] Based on the super-resolution algorithm framework described above, the image processing method provided in this application embodiment may include the following steps 1 to 5.

[0131] Step 1: Portrait Background Perception Module

[0132] An electronic device can acquire a 1K resolution image A and input it into the model of the aforementioned super-resolution algorithm framework. The model first performs image segmentation on the 1K resolution image A using a human detection and instance segmentation model to obtain the specific pixel coordinates of the human and background images. Then, the 1K resolution image A and the 1K resolution human image are upsampled to 4K resolution. Further, the model inputs the 4K resolution image A and the 4K resolution human image into subsequent processing branches.

[0133] Alternatively, the model can first upsample the 1K resolution image A to 4K resolution or higher. Taking 4K as an example, the model can then perform image segmentation on the 4K resolution image A using a human detection and instance segmentation model to obtain the specific pixel coordinates of the person and background images. Further, the model inputs the 4K resolution image A and the 4K resolution person image into subsequent processing branches.

[0134] Step 2, Branch 1: Background Cue Word Generation Module

[0135] To enhance the diversity of background details, the model can input a low-quality 4K resolution image A into a graph-to-text model. Specifically, the model can use the Florence-2 model to provide a detailed description of the subject and background information of image A, such as weather, color tone, architectural style, ethnicity of the subjects, and clothing, to obtain descriptive information about image A.

[0136] Step 3, Branch 1: Diverse Background Enhancement Module

[0137] The model can utilize the descriptive information of image A obtained in step 2 and the high-resolution model with a greater tendency for input value diversity of low-quality 4K resolution image A, such as the SSIM of the high-resolution model being greater than 0.5 and less than 0.7, thereby obtaining a 4K resolution image A with rich background details.

[0138] Step 4, Branch 2: Portrait Preprocessing Module

[0139] The model can use a bilateral filter to preprocess the input 4K resolution human image so that the enhanced image does not excessively enhance the original noise and thus avoids the generation of face artifacts. This can reduce or avoid the over-generation of fine-grained information such as human skin texture caused by ultra-high-definition algorithms in related technologies.

[0140] The parameters of the bilateral filter may include: a filter window diameter of 9, a color space standard deviation of 35, and a space standard deviation of 25.

[0141] Step 5, Branch 2: Consistent Portrait Enhancement Module

[0142] The model can input preprocessed 4K resolution portrait images into a portrait super-resolution algorithm with higher consistency, and the SSIM of this portrait super-resolution algorithm can be greater than 0.9, so that the processed portrait images can maintain high consistency with the original images while enriching portrait details.

[0143] Step 6: Background and Portrait Blending

[0144] The model can fuse the enhanced portrait image and the image A with rich background details according to the pixel positions obtained by the segmentation model. Then, color correction techniques, such as extracting low-frequency information from the original image and high-frequency information from the fused image through wavelet transform, are used to sum the two. This ensures that the enhanced image maintains the same tone as the original image and preserves the detail information of the fused image, resulting in a high-quality 4K resolution image A as the final output.

[0145] In summary, this application proposes a super-resolution algorithm framework that decouples the subject and background to enhance the image quality of AIGC-generated images. This framework satisfies users' expectations for increased detail in background objects while ensuring the controllable generation of fine-grained information for sensitive content in the image, such as human figures. This results in enlarged and enhanced images that better meet users' aesthetic standards.

[0146] Meanwhile, since the diffusion model in related technologies cannot distinguish between facial noise and the original moles on the face in low-resolution images, it will simultaneously enhance the details of the noise and the original moles, resulting in more moles on the face. Therefore, in order to address the problem that the portrait detail enhancement module can easily lead to the over-generation of moles and wrinkles on the face, this application introduces a bilateral smoothing filter to preprocess the face part, so as to smooth the noise and prevent the model from over-enhancing during the enhancement stage. This can effectively reduce the occurrence of facial artifacts, improve the accuracy of the enhancement processing of people, and improve image quality.

[0147] It should be noted that the embodiments of this application only describe the image processing method provided in the field of image super-resolution. Similarly, the image processing method provided in the embodiments of this application can also be used in related fields such as video super-resolution.

[0148] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0149] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0150] Figure 6 A schematic diagram of a possible structure of the image processing apparatus involved in an embodiment of this application is shown. For example... Figure 6 As shown, the image processing device 70 may include: a processing module 71;

[0151] The processing module 71 is used to perform image segmentation processing based on the first image to obtain a first subject image; and to perform first image enhancement processing on the first subject image to obtain a second subject image; and to perform second image enhancement processing on the first image based on the first image and the description information of the first image to obtain a second image; and to perform image fusion processing based on the second subject image and the second image to obtain a third image, wherein the third image includes the background image in the second subject image and the second image.

[0152] In one possible implementation, the processing module 71 is further configured to, after performing image segmentation processing based on the first image to obtain a first subject image, perform upsampling processing on the first image and the first subject image respectively to obtain a fourth image and a third subject image, wherein the resolution of the fourth image is greater than that of the first image, and the resolution of the third subject image is greater than that of the first subject image; specifically, the processing module 71 is configured to perform first image enhancement processing on the third subject image through a first image enhancement model to obtain a second subject image; and perform second image enhancement processing on the fourth image based on the descriptive information of the fourth image and the first image through a second image enhancement model to obtain a second image, wherein the second image enhancement model is used to enhance the image details of the background image in the fourth image.

[0153] In one possible implementation, the image processing apparatus 70 provided in this application embodiment may further include: an acquisition module and a generation module; the acquisition module is used to acquire feature information of the first image based on the first image through a graph-to-text model before performing second image enhancement processing on the first image based on the first image and the description information of the first image to obtain the second image; the generation module is used to generate description information of the first image based on the feature information of the first image, the description information being used to describe the main object and background in the first image.

[0154] In one possible implementation, the processing module 71 is further configured to perform image smoothing processing on the first subject image using a bilateral filter before performing first image enhancement processing on the first subject image to obtain the second subject image, thereby obtaining a smoothed first subject image.

[0155] In one possible implementation, the processing module 71 is specifically used to update the main image in the second image to a second main image to obtain a fifth image; and to perform color correction processing on the fifth image to obtain a third image.

[0156] In the image processing apparatus provided in this application embodiment, the image processing apparatus can segment a first image to obtain a first subject image in the first image. Then, the image processing apparatus can perform a first image enhancement process on the first subject image, and based on the first image and its descriptive information, perform a second image enhancement process on the first image. That is, the image processing apparatus can perform different image enhancements on the subject image and the original image, thus providing a more suitable image enhancement for the subject image. Simultaneously, since the descriptive information of the first image is used during the enhancement process, this descriptive information can be used to improve the ability to restore image details during the image enhancement process. Therefore, the image processing apparatus can perform image fusion using a second subject image with better image enhancement and a second image with better detail restoration to obtain a processed third image. In this way, the consistency of the subject and the restoration of background details can be balanced, thereby improving the image quality of the processed image.

[0157] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0158] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0159] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0160] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0161] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0162] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0163] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0164] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0165] The processor 110 is configured to perform image segmentation processing based on a first image to obtain a first subject image; perform first image enhancement processing on the first subject image to obtain a second subject image; perform second image enhancement processing on the first image based on the first image and its description information to obtain a second image; and perform image fusion processing based on the second subject image and the second image to obtain a third image, wherein the third image includes a background image from the second subject image and the second image.

[0166] Optionally, the processor 110 is further configured to, after performing image segmentation processing based on the first image to obtain a first subject image, perform upsampling processing on the first image and the first subject image respectively to obtain a fourth image and a third subject image, wherein the resolution of the fourth image is greater than that of the first image and the resolution of the third subject image is greater than that of the first subject image; specifically, the processor 110 is configured to perform first image enhancement processing on the third subject image through a first image enhancement model to obtain a second subject image; and perform second image enhancement processing on the fourth image based on the descriptive information of the fourth image and the first image through a second image enhancement model to obtain a second image, wherein the second image enhancement model is used to enhance the image details of the background image in the fourth image.

[0167] Optionally, the processor 110 is further configured to, before performing a second image enhancement process on the first image based on the first image and the description information of the first image to obtain the second image, obtain feature information of the first image based on the first image through a graph-to-text model; and generate description information of the first image based on the feature information of the first image, the description information being used to describe the main object and background in the first image.

[0168] Optionally, the processor 110 is further configured to perform image smoothing processing on the first subject image using a bilateral filter before performing first image enhancement processing on the first subject image to obtain a second subject image, thereby obtaining a smoothed first subject image.

[0169] Optionally, the processor 110 is specifically used to update the main image in the second image to a second main image to obtain a fifth image; and to perform color correction processing on the fifth image to obtain a third image.

[0170] In the electronic device provided in this application embodiment, the electronic device can segment a first image to obtain a first subject image in the first image. Then, the electronic device can perform a first image enhancement process on the first subject image, and perform a second image enhancement process on the first image based on the first image and its descriptive information. That is, the electronic device can perform different image enhancements on the subject image and the original image, thus providing a more suitable image enhancement for the subject image. Simultaneously, since the descriptive information of the first image is used during the enhancement process, the ability to restore image details during the image enhancement process can be improved through this descriptive information. Therefore, the electronic device can perform image fusion using a second subject image with better image enhancement and a second image with better detail restoration to obtain a processed third image. In this way, the consistency of the subject and the restoration of background details can be balanced, thereby improving the image quality of the processed image.

[0171] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0172] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0173] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0174] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0175] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0176] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0177] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0178] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0179] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0180] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0183] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized by, The method includes: Image segmentation is performed on the first image to obtain the first main image; The first subject image is subjected to a first image enhancement process to obtain a second subject image, and based on the first image and its description information, the first image is subjected to a second image enhancement process to obtain a second image. A third image is obtained by performing image fusion processing based on the second subject image and the second image. The third image includes the second subject image and the background image in the second image.

2. The method of claim 1, wherein, After performing image segmentation processing based on the first image to obtain the first subject image, the method further includes: The first image and the first subject image are upsampled to obtain a fourth image and a third subject image, wherein the resolution of the fourth image is greater than that of the first image, and the resolution of the third subject image is greater than that of the first subject image. The step of performing a first image enhancement process on the first subject image to obtain a second subject image, and then performing a second image enhancement process on the first image based on the first image and its description information to obtain a second image, includes: The third subject image is subjected to a first image enhancement process using a first image enhancement model to obtain a second subject image; Using a second image enhancement model, based on the description information of the fourth image and the first image, a second image enhancement process is performed on the fourth image to obtain a second image. The second image enhancement model is used to enhance the image details of the background image in the fourth image.

3. The method according to claim 1 or 2, characterized in that, Before performing a second image enhancement process on the first image based on the first image and its description information to obtain the second image, the method further includes: Based on the first image, the feature information of the first image is obtained using the graph-to-text model. Based on the feature information of the first image, descriptive information of the first image is generated, and the descriptive information is used to describe the main object and background in the first image.

4. The method according to claim 1 or 2, characterized in that, Before performing the first image enhancement process on the first subject image to obtain the second subject image, the method further includes: The first subject image is smoothed by using a bilateral filter to obtain a smoothed first subject image.

5. The method according to claim 1, characterized in that, The image fusion process based on the second main image and the second image to obtain the third image includes: The main image in the second image is updated to the second main image to obtain the fifth image; The fifth image is subjected to color correction processing to obtain the third image.

6. An image processing apparatus, characterized in that, The device includes: a processing module; The processing module is configured to perform image segmentation processing based on a first image to obtain a first subject image; perform a first image enhancement processing on the first subject image to obtain a second subject image; perform a second image enhancement processing on the first image based on the first image and the description information of the first image to obtain a second image; and perform image fusion processing based on the second subject image and the second image to obtain a third image, wherein the third image includes the second subject image and the background image in the second image.

7. The apparatus according to claim 6, characterized in that, The processing module is further configured to, after performing image segmentation processing based on the first image to obtain a first subject image, perform upsampling processing on the first image and the first subject image respectively to obtain a fourth image and a third subject image, wherein the resolution of the fourth image is greater than that of the first image, and the resolution of the third subject image is greater than that of the first subject image; specifically, the processing module is configured to perform first image enhancement processing on the third subject image using a first image enhancement model to obtain a second subject image; and perform second image enhancement processing on the fourth image based on the descriptive information of the fourth image and the first image using a second image enhancement model to obtain a second image, wherein the second image enhancement model is used to enhance the image details of the background image in the fourth image.

8. The apparatus according to claim 6 or 7, characterized in that, The device further includes: an acquisition module and a generation module; The acquisition module is used to acquire feature information of the first image based on the first image and its description information before performing second image enhancement processing on the first image to obtain the second image, by means of a graph-to-text model. The generation module is used to generate descriptive information of the first image based on the feature information of the first image. The descriptive information is used to describe the main object and background in the first image.

9. The apparatus according to claim 6 or 7, characterized in that, The processing module is further configured to perform image smoothing processing on the first subject image using a bilateral filter before performing first image enhancement processing on the first subject image to obtain the second subject image, thereby obtaining the smoothed first subject image.

10. The apparatus according to claim 6, characterized in that, The processing module is specifically used to update the main image in the second image to the second main image to obtain the fifth image; and to perform color correction processing on the fifth image to obtain the third image.

Images