Image processing method and device, terminal and computer readable storage medium

Abstract

The invention discloses an image processing method and device, a terminal and a computer readable storage medium. The image rendering effect can be improved. The method comprises the steps of determining a light source scene corresponding to a to-be-rendered image according to light source information in the to-be-rendered image; according to the light source scene, performing dispersion effect optimization on the light spot reference image corresponding to the to-be-rendered image to obtain a gathered light spot reference image; and rendering the to-be-rendered image based on the gathered light spot reference image to obtain a target rendered image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device, terminal and computer-readable storage medium. Background Art

[0002] At present, in some scenes shot with a large aperture, such as the portrait mode of mobile phone photography, the diffuse spot of the point light source in the rendered image obtained by fusion of multiple frames is usually large, which makes the rendered image often appear blurred, thus affecting the texture and three-dimensional sense of the light spot and reducing the rendering effect of the overall image. Summary of the invention

[0003] The present application hopes to provide an image processing method, device, terminal and computer-readable storage medium, which can improve image rendering effects.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, the present application provides an image processing method, the method comprising:

[0006] Determining a light source scene corresponding to the image to be rendered according to light source information in the image to be rendered;

[0007] According to the light source scene, a diffusion effect is optimized for a light spot reference map corresponding to the image to be rendered to obtain a focused light spot reference map;

[0008] Based on the focused light spot reference image, the image to be rendered is rendered to obtain a target rendered image.

[0009] In a second aspect, the present application provides an image processing device, the device comprising:

[0010] A scene determination module, used to determine a light source scene corresponding to the image to be rendered according to light source information in the image to be rendered;

[0011] An optimization module, configured to optimize the diffusion effect of the light spot reference map corresponding to the image to be rendered according to the light source scene, so as to obtain a focused light spot reference map;

[0012] A rendering module is used to render the image to be rendered based on the focused light spot reference image to obtain a target rendered image.

[0013] Optionally, the optimization module is also used to determine the spot area in the spot reference map; determine the target spot parameters corresponding to the spot area according to the correspondence between the light source scene and the preset light source scene and spot parameters; and optimize the diffusion effect of the spot area according to the target spot parameters to obtain the concentrated spot reference map.

[0014] Optionally, the target light spot parameters include a focusing ratio corresponding to the light spot area, and the optimization module is further used to determine an original area size and an original brightness of the light spot area; based on the focusing ratio, the original area size and the original brightness, the area size and brightness of the light spot area are adjusted to obtain the focused light spot reference image.

[0015] Optionally, the optimization module is also used to determine the roundness coefficient and / or the original area size corresponding to the spot area; when the roundness coefficient is greater than a preset coefficient threshold, and / or the original area size is greater than a preset size threshold, the diffusion effect of the spot area is optimized according to the target spot parameters to obtain the concentrated spot reference image.

[0016] Optionally, the optimization module is further used to determine the original brightness of the light spot area; based on the focusing ratio, the original area size and the original brightness, adjust the area size and brightness of the light spot area to obtain the focused light spot reference image.

[0017] Optionally, the optimization module is also used to determine the target area size corresponding to the spot area based on the original area size and the gathering ratio; determine the target brightness corresponding to the spot area based on the original brightness and the target area size; and perform a gathering adjustment on the spot area based on the gathering ratio and the target brightness to obtain the gathered spot reference image.

[0018] Optionally, the optimization module is further used to determine an initial target brightness corresponding to the spot area according to the original brightness and the target area size; and to normalize the initial target brightness according to a preset brightness curve to obtain the target brightness.

[0019] In some embodiments, the optimization module 12 is also used to determine the center of gravity position corresponding to the light spot area; maintain the center of gravity position, cluster the light spot area according to the clustering ratio, and update the brightness of the clustered light spot area according to the target brightness to obtain the clustered light spot reference image.

[0020] Optionally, the rendering module is also used to determine at least one target channel weight corresponding to the pixels in the image to be rendered in at least one color channel according to the light source scene and the corresponding relationship between the preset light source scene and the channel weight; based on the at least one target channel weight, the at least one color channel is rendered in combination with the focused light spot reference map and the image to be rendered to obtain the target rendered image.

[0021] In some embodiments, the optimization module is also used to render the at least one color channel according to the at least one target channel weight in combination with the focused spot reference map and the image to be rendered, and optimize the diffusion effect of the image to be rendered before obtaining the target rendered image to update the image to be rendered.

[0022] Optionally, the optimization module is also used to determine the target brightness area in the image to be rendered; according to a preset clustering ratio, cluster the target brightness area greater than a preset size threshold to update the image to be rendered; and / or, perform mask extraction of the brightness channel of the image to be rendered, and perform numerical erosion and dilation operations on the obtained brightness channel mask to generate a first mask; generate a second mask by dilating the clustered light spot reference image; and according to the first mask and the second mask, numerically suppress the color channel in the image to be rendered to update the image to be rendered.

[0023] Optionally, the optimization module is further used to perform multiple nonlinear transformations on the spot reference image according to multiple transformation coefficients, and perform mask extraction on the spot reference image after each nonlinear transformation according to a preset brightness threshold among multiple preset brightness thresholds, so as to obtain multiple candidate spot masks; combine the original light source point position coordinates, area size, brightness and color information in the spot reference image, fuse the multiple candidate spot masks, and generate a target spot mask corresponding to the spot reference image; determine the spot area in the spot reference image according to the target spot mask. In a third aspect, the present application provides a terminal, including a memory and a processor; wherein,

[0024] The memory is used to store executable instructions;

[0025] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0026] In a fourth aspect, the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the image processing method provided in an embodiment of the present application.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the image processing method provided by the embodiment of the present application.

[0028] The present application provides an image processing method, device, terminal and computer-readable storage medium, which can determine the light source scene corresponding to the image to be rendered according to the light source information in the image to be rendered, thereby optimizing the diffusion effect of the light spot reference map corresponding to the image to be rendered according to the light source scene, reducing the degree of diffusion of the light spot, and obtaining a concentrated light spot reference map; based on the concentrated light spot reference map, the image to be rendered is rendered to obtain a target rendered image. In this way, the diffused light spots are adaptively concentrated according to the light source scene of the image to be rendered, thereby effectively reducing the degree of diffusion of the light spots during the rendering process and improving the rendering effect of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 1 ;

[0030] Figure 2 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 2 ;

[0031] Figure 3 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 3 ;

[0032] Figure 4 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 4 ;

[0033] Figure 5 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 5 ;

[0034] Figure 6 Schematic diagram of the image processing method provided in the embodiment of the present application Figure 6 ;

[0035] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] At present, for images taken with a large aperture, such as the portrait mode of a mobile phone camera, or when rendering images with a large degree of optical blur, the diffuse spots of the point light sources in the rendered images are large. In addition, due to the limitations of the periscope lens structure, the frequency of occurrence of irregular light spots is high and the shape is irregular, which has a great impact on optical imaging. Therefore, the rendered image appears blurred at the edge of the light spot and lacks a clear light spot hierarchy. In high-brightness scenes and complex light source environments, the defects of this light spot are even more obvious, further affecting the visual effect of the overall image. This situation is especially obvious in high-definition display screens, where users expect to see that each light spot has a clear boundary and hierarchy. However, the rendered result often presents a blurred effect, losing the sharpness and clarity that an optical lens should have. This not only affects the texture and three-dimensional sense of the light spot, but also greatly reduces the realism of the overall image. As a result, the image rendering effect is reduced.

[0042] The embodiments of the present application provide an image processing method, device, terminal, and computer-readable storage medium, which can improve the image rendering effect. The image processing method provided in the embodiments of the present application is applied to a terminal. In some embodiments, the terminal may include various types of user terminals with cameras, such as smart phones, multimedia terminal devices, tablet computers, and smart watches, but is not limited thereto.

[0043] The image processing method provided in the embodiment of the present application can be as follows: Figure 1 As shown, including S101-S103, as follows:

[0044] S101 . Determine a light source scene corresponding to the image to be rendered according to light source information in the image to be rendered.

[0045] In an embodiment of the present application, the image to be rendered may include the front-end processing of the image signal processing (ISP) process of the original image captured by the image sensor, such as de-mosaicing, denoising, white balance and color correction, and the resulting image to be rendered requires the next step of background blur rendering.

[0046] Among them, the light source information in the image to be rendered can be determined according to the light spot reference map corresponding to the image to be rendered, and the optical parameters corresponding to the image to be rendered, such as sensitivity (International Standards Organization, ISO), aperture (F value), etc., and at least one of the light spot reference map corresponding to the image to be rendered. Among them, the light spot reference map can be obtained in the ISP processing flow by performing light spot detection, light spot feature extraction and reference map generation on the image to be rendered. The light source point information existing in the image to be rendered is marked in the light spot reference map. Exemplarily, the light source point information may include at least one of the shape, position, size, and brightness of the light source point. In some embodiments, the light source point information in the light spot reference map and at least one of the optical parameters corresponding to the image to be rendered can be used as the light source information of the image to be rendered.

[0047] It can be seen that the light source information represents whether there are obvious light source points in the image to be rendered, as well as the brightness, shape, quantity, size, position and other information of the light source points. Different rendering strategies or rendering methods should be adopted when rendering for different light source information in different images to be rendered. The embodiment of the present application pre-classifies the light source information into scenes, and pre-configures the rendering method or rendering parameters adapted to the light source scene for various light source scenes. For example, if the light source information represents that the brightness of the light source point in the environment is high, it means that the image to be rendered may be a night scene shot, and it needs to be rendered according to the corresponding rendering method or rendering parameters in the night scene. In this way, according to the light source information in the current image to be rendered, the corresponding light source scene can be determined, so as to perform different rendering methods according to the rendering method or rendering parameters corresponding to the light source scene.

[0048] S102 , optimizing the diffusion effect of the light spot reference map corresponding to the image to be rendered according to the light source scene, to obtain a concentrated light spot reference map.

[0049] In an embodiment of the present application, a corresponding rendering method or rendering parameters can be determined according to the light source scene. Here, the rendering method or rendering parameters corresponding to the light source scene represent a spot rendering method or rendering parameters that are adapted to information such as the brightness, quantity, size, and position of the light source points represented by the light source scene.

[0050] In an embodiment of the present application, when rendering an image to be rendered for image effects such as blurring, the light spot reference map and the blur radius map (blur map) output by the depth estimation module are combined for rendering. During the rendering process, the light spot reference map is used to provide information such as the position, brightness, size, shape, etc. of the light spot in the image for the rendering module to perform corresponding rendering processing. In an embodiment of the present application, the light spot reference map corresponding to the rendered image can be optimized for diffusion effect according to the light source scene, so that the light spot area in the light spot reference map can be gathered to reduce diffusion and obtain a gathered light spot reference map. In this way, rendering according to the gathered light spot reference map can reduce the degree of diffusion of the light spot in the rendered image.

[0051] In some embodiments, the spot area in the spot reference map can be determined by mask extraction, connected domain calculation, etc. Here, the number of spot areas can be one or more, and each spot area represents a spot formed by a light source point. According to the correspondence between the light source scene and the preset light source scene and spot parameters, the target spot parameters corresponding to the spot area are determined; according to the target spot parameters, the diffusion effect of the spot area is optimized to obtain a concentrated spot reference map.

[0052] In some embodiments, the spot reference image can be binarized. For example, according to a first preset brightness threshold, each pixel in the spot reference image is binarized, and a value greater than or equal to the first preset brightness threshold is recorded as a first value, and a value less than the first preset brightness threshold is recorded as a second value. Taking the first value as 255 and the second value as 10 as an example, 255 represents pixels with higher brightness in the spot reference image, which may be pixels corresponding to light source points or spots, and 10 represents pixels with lower brightness in the spot reference image. It can be seen that by binarization, a mask corresponding to the spot reference image can be obtained. The mask marks the information of the pixels corresponding to the light source and the spots formed by it in the spot reference image. The mask corresponding to the spot reference image is calculated using a connected domain algorithm, and pixels with higher brightness can be connected into a spot area. The spot area corresponds to the part that needs to be rendered as a spot when the background is blurred.

[0053] In some embodiments, the light spot reference image may be subjected to multiple nonlinear transformations with different coefficients to improve its contrast, and the light spot reference image may be subjected to multiple binarization extractions by setting different brightness thresholds to generate multiple binarization masks. Finally, the multiple binarization masks are fused in combination with the original light source point position coordinates, area size, brightness and color information to generate a mask corresponding to the final light spot reference image. Exemplarily, the above process of determining the light spot area in the light spot reference image may also include:

[0054] According to multiple transformation coefficients, multiple nonlinear transformations are performed on the spot reference image, and according to a preset brightness threshold among multiple preset brightness thresholds, mask extraction is performed on the spot reference image after each nonlinear transformation, so as to obtain multiple candidate spot masks; combining the original light source point position coordinates, area size, brightness and color information in the spot reference image, multiple candidate spot masks are fused to generate a target spot mask corresponding to the spot reference image; according to the target spot mask, the spot area in the spot reference image is determined.

[0055] Among them, after each nonlinear transformation, a binary extraction is performed on the nonlinearly transformed light spot reference image according to a preset brightness threshold to obtain a candidate light spot mask. Through multiple nonlinear transformations with different transformation coefficients and extractions with different preset brightness thresholds, multiple candidate light spot masks are obtained, and the target light spot mask is obtained by fusing multiple candidate light spot masks. In this way, it can be avoided that when a single brightness threshold is used to extract a mask, weaker light spots are missed due to the brightness threshold being too large, or light spots of different light sources that are close to each other are connected into one piece due to the brightness threshold being too small. This further improves the accuracy of the target light spot mask in identifying the light spot area, thereby improving the image rendering effect.

[0056] Exemplarily, the weight corresponding to each candidate spot mask can be determined in combination with the original light source point position coordinates, area size, brightness and color information in the spot reference image, and multiple candidate spot masks can be fused according to the weight corresponding to each candidate spot mask to generate a target spot mask corresponding to the spot reference image.

[0057] Exemplarily, the nonlinear transformation may include a gamma transformation, or other nonlinear transformation methods applicable to image processing, which are not specifically limited here.

[0058] In some embodiments, the light spot parameters corresponding to the preset light source scene can be obtained from empirical values, or multiple light source scenes can be simulated by simulation, and the optimal parameters for light spot rendering of the light source scene can be obtained by debugging under each light source scene, as the preset light spot parameters corresponding to the light source scene, thereby obtaining the corresponding relationship between the preset light source scene and the light spot parameters. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited.

[0059] Here, according to the light source scene corresponding to the image to be rendered, the corresponding spot parameters are determined in the correspondence between the preset light source scene and the spot parameters, as the target spot parameters corresponding to the spot area in the spot reference image. In some embodiments, for multiple spot areas, the target spot parameters corresponding to each spot area can be determined. Exemplarily, the correspondence between the preset light source scene and the spot parameters may include: at least one preset spot parameter corresponding to at least one preset light source point information in each light source scene of multiple light source scenes. Among them, the preset light source point information may include at least one of the brightness, shape and size of the light source point within a preset numerical range representing the light source scene. The preset spot parameters may include at least one parameter representing the brightness, shape and size of the spot. The preset spot parameters corresponding to the preset light source point information represent the optimal spot rendering parameters corresponding to the preset light source point information in a light source scene. That is to say, for multiple preset light source scenes, different preset light source scenes may correspond to different optimal light spot rendering parameters, and for a preset light source scene, different preset light source point information may correspond to different optimal light spot rendering parameters, which are used as the preset light spot parameters corresponding to the preset light source point information under the preset light source scene, and are used to constrain at least one of the brightness, shape and size of the light spot rendered based on the preset light source point information. In this way, according to the light source scene of the image to be rendered, the optimal light spot rendering parameters corresponding to each light spot area in the light spot reference map corresponding to the image to be rendered under the light source scene can be matched, that is, the target light spot parameters corresponding to each light spot area, and the target light spot parameters corresponding to each light spot area are used to optimize the diffusion effect of the light spot area to obtain a concentrated light spot reference map.

[0060] S103 , rendering the image to be rendered based on the focused light spot reference image to obtain a target rendered image.

[0061] In the embodiment of the present application, each spot area in the concentrated spot reference map is a concentrated spot area obtained by optimizing the diffusion effect in S102. Therefore, rendering the image to be rendered based on the concentrated spot reference map can reduce the degree of diffusion of the spot in the obtained target rendered image and improve the rendering effect.

[0062] In some embodiments, taking diffusion rendering as an example, each color channel of the image to be rendered can be diffused, and the image to be rendered after the color channel diffusion is convolved with the focused spot reference map to obtain a target rendered image. Exemplarily, the color channel diffusion can include diffusion of the brightness (Y) channel and the chrominance (U, V) channels respectively.

[0063] In some embodiments, during the background rendering process, the algorithm usually filters out some background noise, while the foreground noise of the image is not filtered. In order to maintain a balance between the foreground and background noise levels, the rendered image can also be processed with noise, that is, appropriate noise is added to the background of the rendered image to further improve the realism of the image. For example, the appropriate noise type and intensity can be selected according to the overall style and texture of the image to make the noise distribution more natural.

[0064] For example, the overall image rendering process of the embodiment of the present application can be as follows: Figure 2 As shown, it includes: obtaining the image to be rendered, calculating the blur radius map of the image to be rendered through the depth estimation module, and the blur radius map records the blur degree information of each area in the image to be rendered. In the rendering process, the image to be rendered is subjected to light source scene judgment, spot reference map processing (equivalent to optimizing the diffusion effect of the spot reference map), Y channel diffusion, UV channel diffusion and YUV noise addition processes to obtain the target rendering image.

[0065] It can be understood that the embodiment of the present application determines the light source scene corresponding to the image to be rendered based on the light source information in the image to be rendered, and then optimizes the diffusion effect of the light spot reference map corresponding to the image to be rendered based on the light source scene, reduces the degree of diffusion of the light spot, and obtains a gathered light spot reference map; based on the gathered light spot reference map, the image to be rendered is rendered to obtain a target rendered image. In this way, the diffused light spots are adaptively gathered based on the light source scene of the image to be rendered, thereby effectively reducing the degree of diffusion of the light spots during the rendering process and improving the rendering effect of the image.

[0066] In some embodiments, the target light spot parameters may include a focusing ratio corresponding to the light spot area. The process of optimizing the diffusion effect of the light spot area according to the target light spot parameters to obtain a focusing light spot reference map may be as follows: Figure 3 As shown, it is implemented through S301-S302 as follows:

[0067] S301, determining the original area size and original brightness of the light spot area.

[0068] In an embodiment of the present application, the original area size of the spot area may include data characterizing the size, such as the original radius or original area of ​​the spot area. The original area size and original brightness of the spot area may be obtained by calculating the size and brightness based on the mask of the spot reference image through a connected domain algorithm.

[0069] S302 , adjusting the area size and brightness of the light spot area based on the focusing ratio, the original area size and the original brightness, and obtaining a focusing light spot reference image.

[0070] In an embodiment of the present application, the original area size of the spot area can be clustered according to the clustering ratio. Since the total brightness energy of the spot area needs to be kept unchanged after clustering, the brightness of the spot area needs to be recalculated based on the area size after clustering, so as to adjust the area size and brightness of the spot area and obtain a clustered spot reference image.

[0071] In some embodiments, the target area size corresponding to the spot area is determined according to the original area size and the convergence ratio; the target brightness corresponding to the spot area is determined according to the original brightness and the target area size; illustratively, the total brightness energy corresponding to the spot area can be obtained according to the product of the original area size and the original brightness, and the target brightness corresponding to the spot area can be determined according to the ratio of the total brightness energy to the target area size. In this way, according to the convergence ratio and the target brightness, the spot area in the spot reference image is converged and adjusted to obtain a converged spot reference image.

[0072] In some embodiments, the center of gravity position corresponding to the spot area can be determined; when the spot area is focused and adjusted, the center of gravity position is maintained, the spot area is focused according to the focusing ratio, and the brightness of the focused spot area is updated according to the target brightness, thereby optimizing the diffusion effect of the spot area and obtaining a focused spot reference image.

[0073] In some embodiments, the spot areas in the spot reference map may be screened first, and the spot areas meeting the gathering conditions may be gathered. For example, the roundness coefficient corresponding to the spot area may be determined; when the roundness coefficient is greater than a preset coefficient threshold, and / or the original area size is greater than a preset size threshold, the gathering ratio corresponding to the spot area is determined according to the target spot parameters and the original area size, and the area size and brightness of the spot area are adjusted according to the gathering ratio and the original brightness to obtain a gathered spot reference map.

[0074] Here, the gathering condition may include: at least one of the roundness coefficient being greater than a preset coefficient threshold, and / or the original area size being greater than a preset size threshold. Wherein, the roundness coefficient being greater than the preset coefficient threshold indicates that the light spot is close to a circle, and the gathering operation in this case has little deformation on the shape of the light spot, and the gathering operation can be implemented. The original area size being greater than the preset size threshold indicates that the degree of dispersion of the light spot area is large, and a gathering operation needs to be implemented. The gathering condition can also be predefined according to actual conditions, and the embodiments of the present application are not limited thereto.

[0075] It can be understood that the light source points are adaptively gathered by the gathering ratio corresponding to the light source scene, the dispersion of the light source points is reduced, and the rendering effect is improved. By gathering the spot area that meets the gathering conditions, targeted optimization can be implemented to reduce the impact of the gathering operation on the shape and further improve the rendering effect.

[0076] In some embodiments, the above process of determining the target brightness corresponding to the spot area based on the original brightness and the target area size may include: determining the initial target brightness corresponding to the spot area based on the original brightness and the target area size; and normalizing the initial target brightness according to a preset brightness curve to obtain the target brightness.

[0077] Among them, determining the initial target brightness corresponding to the spot area according to the original brightness and the target area size may include: obtaining the total brightness energy corresponding to the spot area according to the product of the original brightness and the original area size, and obtaining the initial target brightness corresponding to the spot area according to the ratio of the total brightness energy corresponding to the spot area to the target area size.

[0078] Here, since the original brightness is obtained based on the binary mask of the light spot reference image, the numerical range of the original brightness can be maintained within the binary numerical interval, such as 10-255. It can usually be recorded by a float array. When the brightness value is recalculated according to the size of the target area after gathering, the initial brightness value obtained will exceed the numerical interval. The embodiment of the present application normalizes the initial target brightness according to a preset brightness curve, wherein the preset brightness curve characterizes the correspondence between brightness values ​​in different numerical intervals. Using the preset brightness curve, the initial target brightness can be obtained to have the same numerical interval as the original brightness, so that the brightness of the light source points after gathering can maintain an appropriate hierarchical relationship.

[0079] It should be noted that the preset brightness curve may include a nonlinear curve, in which the mapping coefficient of the high brightness value is different from the mapping coefficient of the low brightness value, so that the brightness value of the gathered light source point can be finely normalized, keeping the original light source brightness level unchanged and improving the realism of the rendering effect.

[0080] In some embodiments, for the process in S103, in the process of rendering the image to be rendered, for the diffusion strategy of different color channels, such as (YUV three channels), it is necessary to consider the fact that the human eye is more sensitive to the brightness channel, appropriately strengthen the diffusion effect of the Y channel, and balance the color rendering of the U and V channels to improve the final rendering effect. Figure 4 The process shown is to determine the corresponding weight of each color channel during rendering according to the light source scene to achieve better rendering effect, as follows:

[0081] S401 , determining at least one target channel weight corresponding to at least one color channel of a pixel in an image to be rendered according to a light source scene and a preset correspondence between the light source scene and the channel weight.

[0082] In an embodiment of the present application, the correspondence between the preset light source scene and the channel weight can be determined in advance by simulation debugging or the like. Here, the correspondence between the preset light source scene and the channel weight represents at least one optimal rendering weight corresponding to at least one color channel under different light source scenes. Exemplarily, the optimal rendering weight can represent the diffusion weight of the enhanced brightness channel and balance the diffusion weights of the U and V channels.

[0083] Here, the rendering of the image to be rendered is performed in pixels. For each pixel to be rendered, the channel weight corresponding to each color channel of the pixel in at least one color channel can be determined according to the correspondence between the preset light source scene and the channel weight, as the target channel weight corresponding to each color channel, thereby determining at least one target channel weight corresponding to the pixel in at least one color channel.

[0084] S402 , rendering at least one color channel according to at least one target channel weight in combination with the focused spot reference image and the image to be rendered to obtain a target rendered image.

[0085] In the embodiment of the present application, for each color channel of the image to be rendered, the image data in the color channel is diffused according to the target channel weight corresponding to the color channel, and the diffused image data is convolved with the focused light spot reference map. According to the above process, at least one color channel is rendered in units of pixels to obtain a target rendered image.

[0086] In some embodiments, since the target rendered image is obtained based on the spot reference map and the image to be rendered, the above embodiment optimizes the diffusion effect of the spot reference map to obtain a focused spot reference map, thereby achieving spot gathering of the spot reference map. The embodiment of the present application can also perform spot gathering on the image to be rendered to achieve alignment with the spot reference map, thereby further improving the rendering effect. Before rendering the target rendered image according to the target rendering parameters and combining the focused spot reference map with the image to be rendered to obtain the target rendered image, the image to be rendered can also be optimized for diffusion effect to update the image to be rendered.

[0087] In some embodiments, optimizing the diffusion effect of the image to be rendered to update the image to be rendered may include:

[0088] Determine a target brightness region in the image to be rendered; and gather target brightness regions greater than a preset size threshold according to a preset gathering ratio to update the image to be rendered.

[0089] Here, the target brightness region may be an image region in the image to be rendered whose brightness value is higher than a preset region brightness threshold. Exemplarily, the Y channel representing the brightness in the image to be rendered may be subjected to binary mask extraction according to a second preset brightness threshold, and a region in the image to be rendered with a brightness greater than or equal to the second preset brightness threshold may be determined as the target brightness region. The target brightness region may be one or more. According to a preset gathering ratio, target brightness regions greater than a preset size threshold are gathered, thereby updating the image to be rendered and optimizing the diffusion effect of the image to be rendered.

[0090] For example, Figure 5 As shown, the Y channel of the image to be rendered in the input data is processed Figure 2 For example, the Y channel map of the image to be rendered can be binarized with (252, 255) to obtain the mask corresponding to the image to be rendered, and the target brightness area is determined according to the mask corresponding to the image to be rendered. The area with a blur radius greater than 4 (equivalent to the target brightness area greater than a preset size threshold) is gathered, thereby updating the Y channel map in the image to be rendered, and the image to be rendered is updated. For the light spot reference map in the input data, the light spot reference map is binarized. For example, the light spot reference map is subjected to (10, 255) binary mask extraction to obtain the mask information of the light source point. The connected domain algorithm is used to calculate the connected domain corresponding to each light source point, that is, the brightness, area (equivalent to the original area size), shape, and center of gravity position of the spot area. When the roundness coefficient of the connected domain shape roundness detection is greater than 0.58 (equivalent to the roundness coefficient greater than the roundness coefficient threshold), and the blur radius is greater than 4 (equivalent to the original size area greater than the preset size threshold), the gathering ratio corresponding to the connected domain, that is, the spot area, is determined according to the input light source scene. While keeping the center of gravity position unchanged, the spot area is proportionally gathered, and its area brightness is recalculated, and then the brightness of the gathered area is normalized according to the preset brightness curve to achieve the gathering of the spot area; when the roundness coefficient of the connected domain shape roundness detection is less than or equal to 0.5, or the blur radius is less than or equal to 4, the spot area is kept unchanged and no gathering processing is performed, thereby obtaining a gathered spot reference map. According to the gathered spot reference map, the updated image to be rendered and the blur radius map in the input data are diffusely rendered using the point spread function (PSF) to generate the final target rendered image.

[0091] At present, when shooting close-up portraits, the degree of diffusion of the rendered input image tends to increase. This is mainly because the distance between the light source and the lens is shortened, making the light spot in the image more obvious and complex. If only the light spot reference map is optimized, the rendered image may have unnatural halos in the edge area of ​​the light spot. This halo phenomenon will make the rendered image appear false, lacking in real optical effects, and not in line with the visual habits of the human eye. In order to further improve the light spot texture of the rendered image, the above-mentioned optimization of the diffusion effect of the image to be rendered to update the image to be rendered may include:

[0092] The brightness channel in the image to be rendered is masked and the extracted brightness channel mask is numerically eroded and expanded to generate a first mask; the second mask is generated by expanding the gathered light spot reference image; the color channel in the image to be rendered is numerically suppressed according to the first mask and the second mask to update the image to be rendered.

[0093] In some embodiments, the process of generating the second mask by dilating the focused spot reference image may include: dilating the focused spot reference image to obtain an expanded spot reference image; downsampling and dilating the expanded spot reference image to generate the second mask.

[0094] In some embodiments, numerically suppressing the color channels in the image to be rendered according to the first mask and the second mask to update the image to be rendered may include:

[0095] Convert the image to be rendered in the first color domain to the second color domain; perform color channel erosion on the image to be rendered in the second color domain, and convert the image to be rendered after the color channel erosion to the first color domain to obtain a preprocessed image to be rendered; perform numerical suppression on at least one color channel in the preprocessed image to be rendered according to the first mask and the second mask, so as to update the image to be rendered.

[0096] Exemplarily, the first color domain may include a YUV domain, and the second color domain may include an RGB domain. This embodiment of the present application is not specifically limited.

[0097] For example, the above process can be performed as follows Figure 6 As shown, including:

[0098] S601, treat the Y channel of the rendered image Figure 2 Value it and get the Y channel mask.

[0099] In S601, the Y channel of the rendered image is treated Figure 2 For example, the Y channel map of the rendered image is subjected to binary extraction of (130, 255) to obtain a Y channel mask (equivalent to a brightness channel mask).

[0100] S602: Eroding and dilating the Y channel mask to generate a first mask.

[0101] In S602, the Y channel mask is eroded and expanded in sequence to generate a first mask.

[0102] S603 , performing expansion, downsampling and expansion operations on the gathered light spot reference image to obtain a second mask.

[0103] S604 , eroding the image to be rendered to obtain a preprocessed image to be rendered.

[0104] In S604, the image to be rendered is converted from the YUV domain to the RGB domain, three-channel erosion is performed, and then converted to the YUV domain to obtain a pre-processed image to be rendered.

[0105] S605, YUV three-channel backfill.

[0106] In S605, the Y channel in the preprocessed image to be rendered is numerically suppressed according to the first mask and the second mask to reduce the diffusion effect of the light spot, and the U channel and the V channel are numerically suppressed according to the first mask and the second mask to maintain the color balance of the image. In this way, the brightness and chromaticity information of the image are finely controlled, and the image to be rendered is updated to achieve the visual effect of the light spot shrinking.

[0107] It can be understood that the embodiment of the present application optimizes the design of the three YUV channels of the image to be rendered to suppress unnatural halo phenomena, and ultimately generates a spot rendering image with sharp edges, distinct brightness levels, and close to the optical blur effect, thereby achieving the effect of retracting the diffuse light source.

[0108] The present application embodiment provides an image processing device 1, such as Figure 7 As shown, including:

[0109] The scene determination module 11 is used to determine the light source scene corresponding to the image to be rendered according to the light source information in the image to be rendered;

[0110] An optimization module 12 is used to optimize the diffusion effect of the light spot reference map corresponding to the image to be rendered according to the light source scene to obtain a concentrated light spot reference map;

[0111] The rendering module 13 is used to render the image to be rendered based on the focused light spot reference image to obtain a target rendered image.

[0112] In some embodiments, the optimization module 12 is also used to determine the spot area in the spot reference map; determine the target spot parameters corresponding to the spot area according to the correspondence between the light source scene and the preset light source scene and spot parameters; and optimize the diffusion effect of the spot area according to the target spot parameters to obtain the concentrated spot reference map.

[0113] In some embodiments, the target light spot parameters include a convergence ratio corresponding to the light spot area, and the optimization module 12 is further used to determine an original area size and an original brightness of the light spot area; based on the convergence ratio, the original area size, and the original brightness, the area size and brightness of the light spot area are adjusted to obtain the converged light spot reference image.

[0114] In some embodiments, the optimization module 12 is also used to determine the roundness coefficient and / or the original area size corresponding to the spot area; when the roundness coefficient is greater than a preset coefficient threshold, and / or the original area size is greater than a preset size threshold, the diffusion effect of the spot area is optimized according to the target spot parameters to obtain the concentrated spot reference image.

[0115] In some embodiments, the optimization module 12 is further used to determine the original brightness of the light spot area; based on the focusing ratio, the original area size and the original brightness, the area size and brightness of the light spot area are adjusted to obtain the focused light spot reference image.

[0116] In some embodiments, the optimization module 12 is also used to determine the target area size corresponding to the spot area based on the original area size and the gathering ratio; determine the target brightness corresponding to the spot area based on the original brightness and the target area size; and perform a gathering adjustment on the spot area based on the gathering ratio and the target brightness to obtain the gathered spot reference image.

[0117] In some embodiments, the optimization module 12 is further used to determine the initial target brightness corresponding to the spot area according to the original brightness and the target area size; and to normalize the initial target brightness according to a preset brightness curve to obtain the target brightness.

[0118] In some embodiments, the optimization module 12 is also used to determine the center of gravity position corresponding to the light spot area; maintain the center of gravity position, cluster the light spot area according to the clustering ratio, and update the brightness of the clustered light spot area according to the target brightness to obtain the clustered light spot reference image.

[0119] In some embodiments, the rendering module 13 is also used to determine at least one target channel weight corresponding to the pixels in the image to be rendered in at least one color channel according to the light source scene and the corresponding relationship between the preset light source scene and the channel weight; according to the at least one target channel weight, the at least one color channel is rendered in combination with the focused light spot reference map and the image to be rendered to obtain the target rendered image.

[0120] In some embodiments, the optimization module 12 is also used to render the at least one color channel according to the at least one target channel weight in combination with the focused spot reference map and the image to be rendered, and optimize the diffusion effect of the image to be rendered before obtaining the target rendered image to update the image to be rendered.

[0121] In some embodiments, the optimization module 12 is also used to determine the target brightness area in the image to be rendered; according to a preset clustering ratio, cluster the target brightness area greater than a preset size threshold to update the image to be rendered; and / or, perform mask extraction of the brightness channel of the image to be rendered, and perform numerical erosion and dilation operations on the obtained brightness channel mask to generate a first mask; generate a second mask by dilating the clustered light spot reference image; and according to the first mask and the second mask, numerically suppress the color channel in the image to be rendered to update the image to be rendered.

[0122] In some embodiments, the optimization module 12 is further used to perform multiple nonlinear transformations on the spot reference image according to multiple transformation coefficients, and perform mask extraction on the spot reference image after each nonlinear transformation according to a preset brightness threshold among multiple preset brightness thresholds, so as to obtain multiple candidate spot masks; combine the original light source point position coordinates, area size, brightness and color information in the spot reference image, fuse the multiple candidate spot masks, and generate a target spot mask corresponding to the spot reference image; determine the spot area in the spot reference image according to the target spot mask.

[0123] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0124] The embodiment of the present application also provides a terminal, Figure 8 This is an optional structural diagram of a terminal provided in an embodiment of the present application. Figure 8As shown, the terminal 3 includes: a memory 32 and a processor 33. The memory 32 and the processor 33 are connected via a communication bus 34; the memory 32 is used to store executable instructions; the processor 33 is used to implement the image processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory 32.

[0125] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by the above-mentioned processor, the above-mentioned processor will execute the image processing method provided by the embodiment of the present application.

[0126] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0127] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0128] As examples, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As examples, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0130] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] The above is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Determining a light source scene corresponding to the image to be rendered according to light source information in the image to be rendered; According to the light source scene, a diffusion effect is optimized for a light spot reference map corresponding to the image to be rendered to obtain a focused light spot reference map; Based on the focused light spot reference image, the image to be rendered is rendered to obtain a target rendered image.

2. The method according to claim 1, characterized in that The step of optimizing the diffusion effect of the light spot reference map corresponding to the image to be rendered according to the light source scene to obtain a focused light spot reference map includes: Determine a light spot area in the light spot reference image; Determining target light spot parameters corresponding to the light spot area according to the correspondence between the light source scene and the preset light source scene and light spot parameters; According to the target light spot parameters, the diffusion effect of the light spot area is optimized to obtain the focused light spot reference image.

3. The method according to claim 2, characterized in that The step of optimizing the diffusion effect of the light spot area according to the target light spot parameters to obtain the focused light spot reference map includes: Determine the roundness coefficient and / or original area size corresponding to the light spot area; When the roundness coefficient is greater than a preset coefficient threshold, and / or the original area size is greater than a preset size threshold, the diffusion effect of the spot area is optimized according to the target spot parameters to obtain the focused spot reference image.

4. The method according to claim 2 or 3, characterized in that: The target light spot parameter includes a gathering ratio corresponding to the light spot area, and the step of optimizing the diffusion effect of the light spot area according to the target light spot parameter to obtain the gathering light spot reference map includes: Determining the original brightness of the light spot area; Based on the focusing ratio, the original area size and the original brightness, the area size and brightness of the light spot area are adjusted to obtain the focused light spot reference image.

5. The method according to claim 4, characterized in that The adjusting the area size and brightness of the light spot area based on the focusing ratio, the original area size and the original brightness to obtain the focusing light spot reference image includes: Determining a target area size corresponding to the spot area according to the original area size and the gathering ratio; Determining a target brightness corresponding to the light spot area according to the original brightness and the target area size; According to the focusing ratio and the target brightness, the light spot area is focused and adjusted to obtain the focused light spot reference image.

6. The method according to claim 5, characterized in that The determining, according to the original brightness and the target area size, the target brightness corresponding to the spot area includes: Determining an initial target brightness corresponding to the light spot area according to the original brightness and the target area size; The initial target brightness is normalized according to a preset brightness curve to obtain the target brightness.

7. The method according to claim 5, characterized in that The step of performing focusing adjustment on the light spot area according to the focusing ratio and the target brightness to obtain the focusing light spot reference image includes: Determine the center of gravity position corresponding to the light spot area; The center of gravity position is maintained, the light spot area is clustered according to the clustering ratio, and the brightness of the clustered light spot area is updated according to the target brightness to obtain the clustered light spot reference image.

8. The method according to any one of claims 1 to 3, or any one of claims 5 to 7, characterized in that: The step of rendering the image to be rendered based on the focused light spot reference image to obtain a target rendered image comprises: Determine at least one target channel weight corresponding to at least one color channel of a pixel in the image to be rendered according to the light source scene and a correspondence between a preset light source scene and a channel weight; According to the at least one target channel weight, the at least one color channel is rendered in combination with the focused spot reference map and the image to be rendered to obtain the target rendered image.

9. The method according to claim 8, characterized in that Before rendering the at least one color channel according to the at least one target channel weight in combination with the focused spot reference map and the image to be rendered to obtain the target rendered image, the method further includes: The diffusion effect of the image to be rendered is optimized to update the image to be rendered.

10. The method according to claim 9, characterized in that The step of optimizing the diffusion effect of the image to be rendered to update the image to be rendered includes: Determining a target brightness area in the image to be rendered; According to a preset aggregation ratio, the target brightness area greater than a preset size threshold is aggregated to update the image to be rendered; and / or, Extracting a mask of a brightness channel of the image to be rendered, and performing numerical erosion and dilation operations on the obtained brightness channel mask to generate a first mask; Generate a second mask by dilating the focused light spot reference image; According to the first mask and the second mask, the color channels in the image to be rendered are numerically suppressed to update the image to be rendered.

11. The method according to claim 2 or 3, or any one of claims 5 to 7, or claim 9 or claim 10, characterized in that: Determining the spot area in the spot reference image comprises: According to the plurality of transformation coefficients, the light spot reference image is subjected to a plurality of nonlinear transformations, and according to a preset brightness threshold among a plurality of preset brightness thresholds, a mask is extracted from the light spot reference image after each nonlinear transformation, so as to obtain a plurality of candidate light spot masks; Combine the original light source point position coordinates, area size, brightness and color information in the light spot reference image, fuse multiple candidate light spot masks, and generate a target light spot mask corresponding to the light spot reference image; According to the target light spot mask, a light spot area in the light spot reference image is determined.

12. An image processing device, characterized in that: The device comprises: A scene determination module, used to determine a light source scene corresponding to the image to be rendered according to light source information in the image to be rendered; An optimization module, configured to optimize the diffusion effect of the light spot reference map corresponding to the image to be rendered according to the light source scene, so as to obtain a focused light spot reference map; A rendering module is used to render the image to be rendered based on the focused light spot reference image to obtain a target rendered image.

13. A terminal, characterized in that: comprising a memory and a processor; wherein, The memory is used to store executable instructions; The processor is used to implement the method according to any one of claims 1-11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause a processor to execute and implement any method according to claims 1-11.

15. A computer program product, characterized in that The method comprises a computer program or an instruction, which implements the method according to any one of claims 1 to 11 when the computer program or the instruction is executed by a processor.

Images