Image processing method, electronic equipment and storage medium

Abstract

The invention provides an image processing method, electronic equipment and a storage medium. The method comprises the following steps: in a scene of performing image acquisition by using a target application, triggering background lighting processing on a to-be-displayed first preview image through a first control on a photographing preview interface to obtain a second preview image; and then displaying the second preview image on a photographing preview interface. If the trigger operation on the first control is detected, the collected preview image is not directly displayed, but the preview image to be displayed is firstly subjected to background lighting processing, and then the preview image subjected to lighting processing is displayed on the photographing preview interface. In this way, it can be achieved that in the photographing preview process, it is triggered that the lighting effect is added to the background area in the preview image to highlight the line contour of the photographed subject, a user can determine whether the background lighting effect meets the expectation or not through the preview image displayed on the photographing preview interface, the image lighting processing process is preposed, and the user experience is improved. Furthermore, only the shot image after the lighting processing can be stored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent terminals, and in particular to an image processing method, an electronic device, and a storage medium. BACKGROUND

[0002] With the continuous upgrading of the functions of electronic devices, the application of cameras in electronic devices is becoming more and more widespread to meet the needs of users for taking pictures. On the one hand, for daily picture-taking scenarios, in order to highlight the line profile of the photographed subject and enhance the sense of levels of the picture, it is usually necessary to add a certain lighting effect on the image background, and therefore, there is a need for lighting processing of images. On the other hand, for the application scenario of training a model based on sample images, it is necessary to have images with different background lighting effects as sample images to ensure the diversification of the lighting effects of sample images, and at this time, there is also a need for lighting processing of images. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present application provide an image processing method, an electronic device, and a storage medium. In the method, it is possible to trigger the addition of a background lighting effect to a background region in a preview image in a picture-taking preview process, and display the preview image after lighting processing in a picture-taking preview interface, so that the user can determine whether the background lighting effect meets the expectation through the preview image displayed in the picture-taking preview interface, thereby realizing the front positioning of the image lighting processing process, and further, in the case where the lighting effect of the preview image displayed on the picture-taking preview interface meets the expectation, it is possible to selectively store only the captured image after lighting processing.

[0004] In a first aspect, the embodiments of the present application provide an image processing method. The method is applied to an electronic device and includes: displaying a first preview interface of a target application; the target application has an image acquisition function, and the first preview interface includes a first control; in response to a first operation on the first control, displaying a second preview interface of the target application; the second preview interface includes a second preview image, and the second preview image is obtained by performing lighting processing on a background region of a target object in a first preview image; and the first preview image is a frame of preview image acquired by the electronic device.

[0005] For example, the electronic device can be a mobile phone, laptop, tablet, etc. The first interface can be a photo preview interface; the target application can be a camera application or any third-party application with shooting capabilities, such as a chat application or a short video application. The target object can be the target shooting subject mentioned in the following embodiments. The target object can be a target portrait or other shooting object, such as an animal, building, etc. The first control can be the lighting control mentioned in the following embodiments, and the first parameter information can be the light source parameter information mentioned in the following embodiments. The light source parameter information can be a default value, a user-set parameter value, or a parameter value automatically determined based on user input information.

[0006] For example, if the first preview image includes multiple subjects, all subjects can be used as the target subjects, or some subjects can be used as the target subjects; the background area can include the image area in the first preview image other than the target subjects and the subject edge areas of the target subjects.

[0007] For example, the process of background lighting processing on the first preview image may include: performing subject segmentation on the first preview image to obtain subject contour information, determining the target illumination distribution template, determining the illumination coefficient distribution map, multiplying the illumination distribution template and the illumination coefficient distribution map element by element to obtain an illumination weighted rendering map, and performing pixel value weighted calculation processing based on the first preview image and the illumination weighted rendering map.

[0008] Alternatively, the image lighting process can involve lighting only the background area in the first preview image. Specifically, this involves determining a target illumination distribution template, identifying the background area based on the contour position information of the target subject, setting the illumination intensity of non-background areas in the target illumination distribution template to zero to obtain a illumination mask, and then performing image fusion processing based on the first preview image and the illumination mask to obtain a second preview image. However, to improve the realism of the background lighting effect, it is also possible to apply lighting to both the background area and the edge area of ​​the target subject in the first preview image, i.e., adding lighting effects not only to the background area but also to the edge area of ​​the target subject.

[0009] In cases where lighting effects are added to both the background area and the edge area of ​​the target subject, the following steps are required: First, determine the target lighting distribution template; then, determine the background area and the edge area of ​​the target subject based on the outline position information of the target subject; next, determine the lighting coefficient distribution map based on the lighting coefficient variation curves of the background area and the edge area of ​​the subject; then, perform element-wise multiplication calculations based on the target lighting distribution template and the lighting coefficient distribution information to obtain a lighting weighted rendering map; finally, perform image fusion processing based on the lighting weighted rendering map and the first preview image to obtain the second preview image.

[0010] In this scenario, when using the target application for image acquisition, the first control on the photo preview interface can trigger lighting processing on the first preview image to be displayed, resulting in a second preview image; this second preview image is then displayed on the photo preview interface. In other words, as the camera continuously acquires preview images, the preview images displayed on the photo preview interface are refreshed in real time, with multiple frames displayed sequentially. If a trigger operation on the first control is detected, the next frame to be displayed is not directly shown on the photo preview interface; instead, the preview image to be displayed is first lit, and then the lit preview image is displayed on the photo preview interface. Based on this, it is possible to trigger the addition of lighting effects to the background area of ​​the preview image during the photo preview process and display the lit preview image on the photo preview interface. This allows users to determine whether the background lighting effect meets expectations through the preview image displayed on the photo preview interface, thus enabling the image lighting process to be pre-processed. Furthermore, if the lighting effect of the preview image displayed on the photo preview interface meets expectations, only the lit image can be selectively stored.

[0011] According to the first aspect, the method further includes, before displaying a second preview interface of the target application, displaying a third preview interface of the target application; the third preview interface includes a second control; in response to a second operation on the second control, determining at least one target object from a plurality of captured objects in the first preview image; and a background area is determined based on the contour information of the at least one target object.

[0012] Each second control corresponds to an object identifier of a shooting object, and the target object is the shooting object corresponding to the object identifier selected by the user.

[0013] For example, the third preview interface includes not only a preview area but also controls for indicating multiple objects contained in the preview image; the outline information may be the outline position information mentioned in the embodiments below. The object identifier may be an identifier used to characterize the type of the object being photographed, such as a portrait, a tree, etc., or it may be a preset number identifier, such as object 1 and object 2, etc.

[0014] In this way, after detecting the user's image lighting trigger operation, the object identifier of the shooting object is displayed to the user on the preview interface of the target application. The user can select which shooting objects as target objects according to actual needs, so that the user can selectively decide which areas in the image as background areas, thus meeting the user's personalized needs for background lighting effects.

[0015] According to the first aspect, in this method, before displaying the second preview interface of the target application, the method further includes: displaying a first settings interface and, in response to a third operation performed on the first settings interface, determining first parameter information; and performing lighting processing on the background area of ​​the target object in the first preview image according to the first parameter information to obtain a second preview image.

[0016] For example, the first setting interface includes at least one parameter setting control, each parameter setting control being used to set lighting parameters; the first setting interface may be the lighting parameter setting interface mentioned in the following embodiments; the parameter setting control may be a control used to set the value of a certain lighting parameter; the first parameter information includes the parameter values ​​corresponding to multiple lighting parameters respectively.

[0017] In this way, after detecting the trigger operation of the lighting control, a lighting parameter setting interface is provided to the user. The user can set the value of the image lighting parameter according to the actual needs, thereby deciding which light distribution templates to use as the target light distribution templates, so as to meet the user's personalized needs for background lighting effects.

[0018] According to the first aspect, or any implementation of the first aspect above, in this method, the second preview interface further includes a third control; after displaying the second preview interface of the target application, the method further includes: in response to a fourth operation on the third control, acquiring a first captured image, and performing lighting processing on the background area of ​​the target object in the first captured image according to the first parameter information to obtain a second captured image; and storing the second captured image.

[0019] The second control can be a camera control. In a camera shooting scenario, if a user click on the lighting control is detected, the preview image needs to be lit before the user clicks the camera control, and the target lighting image (i.e., the second preview image) corresponding to the preview image needs to be displayed on the camera preview interface. After the user clicks the camera control, the captured image needs to be lit, and the target lighting image (i.e., the second captured image) corresponding to the captured image needs to be stored. The process of lighting the captured image can be referred to the lighting process of the first preview image described above, and will not be repeated here.

[0020] In this way, if the background lighting effect of the target lighting image (i.e., the second preview image) displayed in the photo preview interface meets the user's expectations, the user can click the photo control to trigger the capture of the image through the camera. Then, the captured image is first processed with lighting to obtain the target lighting image (i.e., the second captured image) corresponding to the captured image, and the target lighting image is stored so that the processed captured image can be viewed in the gallery application. At this time, the captured image without lighting processing can be stored directly, instead of the target lighting image with the expected background lighting effect, thus saving the user the step of deleting the captured image without lighting processing to free up memory.

[0021] According to the first aspect, or any implementation of the first aspect above, in this method, the second preview interface further includes a fourth control; after displaying the second preview interface of the target application, the method further includes: determining second parameter information in response to a fifth operation on the fourth control; performing lighting processing on the background area of ​​the target object in the third preview image according to the second parameter information to obtain a fourth preview image; the third preview image is a frame preview image captured by the electronic device; and displaying the fourth preview image on the second preview interface.

[0022] The process of determining the second parameter information includes: displaying a human-computer interaction interface; the human-computer interaction interface including a fifth control; responding to a sixth operation on the fifth control to obtain user input information; the user input information being used to describe the expected lighting effect; and matching light source parameters based on the user input information to determine the second parameter information.

[0023] For example, the second parameter information can be the adjusted light source parameter information; the third control can be a control used to trigger relighting, and the third control can be the same as or different from the first control mentioned above. For example, if the third control is the same as the first control, then the second parameter information can be determined in response to the adjusted parameter value set by the user in the lighting parameter setting interface; or, if the third control is different from the first control, the third control can be a newly added human-computer interaction control, and correspondingly, the second parameter information can also be obtained by matching parameter values ​​based on the description information of the desired lighting effect input by the user; wherein, the acquisition time of the third preview image is later than the acquisition time of the first preview image.

[0024] In this way, if the background lighting effect of the target lighting image displayed in the photo preview interface does not meet the user's expectations, the user can trigger the adjustment of the first parameter information through the third control to obtain the second parameter information, and then use the second parameter information to process the lighting of the preview image to obtain a new target lighting image. This process continues until the lighting effect of the target lighting image meets the user's expectations. This allows for timely adjustment of the background lighting effect during the photo preview process, ensuring that the image stored in the gallery is the target lighting image that meets the user's expectations. This saves the user the steps of deleting already lit images in the gallery application and triggering image relighting. Furthermore, in cases where the user triggers image relighting, it may be because the user is unaware of the correspondence between lighting parameter values ​​and lighting effects, thus making it impossible to determine the accuracy of the set lighting parameters. Therefore, by adding a third control different from the first control, a human-computer interaction interface is displayed after the user's trigger operation on the third control is detected. The user can then input the desired lighting effect description through the information input controls on the human-computer interaction interface. Thus, the adjusted light source parameter information can be obtained by matching parameter values ​​based on the user's input description of the desired lighting effect. For users who are unaware of the correspondence between lighting parameter values ​​and lighting effects, they only need to describe the desired lighting effect without needing to pay attention to the specific values ​​of each lighting parameter.

[0025] Furthermore, after the fourth preview image is displayed on the second preview interface, the process further includes: in response to the triggering operation of the camera control on the second preview interface, acquiring a third captured image, and performing background lighting processing on the third captured image according to the second parameter information to obtain the fourth captured image, and storing the fourth captured image. Thus, if the lighting effect of the target lighting image (i.e., the fourth preview image) with adjusted lighting parameters displayed on the camera preview interface meets the user's expectations, the user can trigger the acquisition of a captured image by clicking the camera control; then, the captured image is first processed with lighting to obtain the target lighting image (i.e., the fourth captured image) corresponding to the captured image, and this target lighting image is stored; at this time, the captured image without lighting processing is not stored, but the target lighting image with the expected lighting effect is directly stored, thus saving the user the step of deleting the captured image without lighting processing to free up memory.

[0026] According to the first aspect, or any implementation of the first aspect above, the method further includes: displaying a first interface of a gallery application; the first interface includes a first image and a sixth control, the first image being an image selected by the user; in response to a seventh operation on the sixth control, displaying a second settings interface; in response to an eighth operation performed on the second settings interface, determining third parameter information; performing lighting processing on the background area of ​​a target object in the first image according to the third parameter information to obtain a second image, and displaying the second image on the second interface of the gallery application.

[0027] For example, the first interface can be the photo display interface of a gallery application, the first image can be the gallery image to be lit as mentioned in the embodiments below, and the fourth control can be a lighting control, similar to the first control described above; the second image can be the third lit image mentioned in the embodiments below, and the second interface can be the image editing interface of a gallery application. The first image can be an image without lighting processing or an image that has already been lit.

[0028] For example, if the first image is an image that has already been illuminated, the first image can be restored first to obtain a corresponding initial image, and then the initial image can be illuminated to obtain the second image. If the corresponding initial image is stored during the illumination process to obtain the first image, then the initial image corresponding to the first image can be directly retrieved from memory, and then the initial image can be illuminated to obtain the second image. This application does not limit this.

[0029] In addition, a lighting parameter setting interface (i.e., the second settings interface) can also be provided for the user during the process of lighting images in the gallery application. The specific implementation process can be referred to the setting process of the first parameter information mentioned above, and will not be repeated here. Furthermore, the lighting process for the first image can be referred to the lighting process for the first preview image mentioned above, and will not be repeated here.

[0030] In this way, users can trigger background lighting processing not only during the photo preview process but also on any image within the gallery application, ensuring flexibility in triggering image lighting processing. Specifically, for scenarios involving lighting images within the gallery application, for example, if a user is dissatisfied with the lighting effect of a stored target image, they can trigger relighting of that image; or, for instance, a user can trigger lighting processing on images transferred from other devices and stored in the gallery.

[0031] Additionally, if the lighting effect of the target lighting image displayed on the second interface meets the user's expectations, the user can trigger the saving of the target lighting image corresponding to the selected image through the save control, so that the target lighting image can be viewed at any time from the gallery application.

[0032] According to the first aspect, or any implementation of the first aspect above, in this method, the background area of ​​the target object in the first preview image is illuminated according to the first parameter information to obtain the second preview image, including: performing subject segmentation processing on the first preview image to obtain the first contour information of the target object in the first preview image; determining the target illumination template from the illumination template set based on the first parameter information; determining the background area and the subject edge area according to the first contour information, and determining the illumination coefficient distribution information of the first preview image according to the illumination coefficient of the background area and the illumination coefficient of the subject edge area; obtaining the illumination weighted rendering map according to the target illumination template and the illumination weighted rendering map; and performing image fusion processing on the first preview image and the illumination weighted rendering map to obtain the second preview image.

[0033] For example, the first contour information may be the original contour information mentioned in the following embodiments, which may be determined based on the contour position information obtained by subject segmentation of the first preview image; the target illumination template may be the target illumination distribution template mentioned in the following embodiments; and the illumination coefficient of the subject edge region may be the illumination coefficient variation curve mentioned in the following embodiments.

[0034] In this process, the background lighting is first processed by selecting a target lighting distribution template corresponding to the light source parameter information from a preset set of lighting distribution templates; then determining the contour information of the target subject; and finally, combining the contour information of the target subject with the determination of the subject edge region in the image. Next, based on the illumination coefficient change curve of the subject edge region, illumination coefficient distribution information is automatically generated. Then, a weighted illumination rendering map is generated based on the target lighting distribution template and the illumination coefficient distribution information, and image fusion processing is performed based on the first preview image and the weighted illumination rendering map to obtain the target lighting image. This not only adds lighting effects to the background area but also to the critical area between the background area and the target subject (which can be called the subject edge region), giving the subject edge region a gradual lighting effect and making the lighting changes between the subject and the background more natural, thereby improving the visual harmony of the image lighting effect for the user.

[0035] According to the first aspect, or any implementation of the first aspect above, in this method, the first parameter information includes the light source direction; determining the background region and the subject edge region based on the first contour information includes: if the light source direction is a front-backlight direction, then determining the second contour information of the target object based on the first contour information, and determining the background region and the subject edge region based on the first contour information and the second contour information; the first contour information is used to characterize the original contour position of the target object, and the second contour information is used to characterize the inner contour position of the target object; if the light source direction is a side-backlight direction, then determining the third contour information of the target object based on the first contour information and the light source direction, and determining the background region and the subject edge region based on the first contour information and the third contour information; the third contour information is used to characterize the projected contour position of the target object along the light source direction.

[0036] For example, the first contour information can be the original contour information mentioned in the following embodiments. The original contour information can be determined based on the contour position information obtained by segmenting the first preview image. The second contour information can be the inner edge contour information mentioned in the following embodiments, that is, the inner circle contour position can be the inner edge contour mentioned in the following embodiments. The third contour information can be the projected edge contour information mentioned in the following embodiments. The projected edge contour of the target subject can be determined based on the spatial angle between the center position of the simulated light source and the center position of the subject. That is, the target subject is projected along the direction of illumination from the center position of the light source to the center position of the subject to obtain the projected edge contour.

[0037] In this way, the process of determining the edge area of ​​the subject is differentiated between front backlight and side backlight. For front backlight, the original outline and inner edge outline of the subject are considered to determine the edge area; while for side backlight, the original outline and projected edge outline of the subject are considered to determine the edge area. This improves the accuracy of determining the edge area and ensures the authenticity of the lighting effect on the edge area.

[0038] According to the first aspect, or any implementation of the first aspect above, in this method, before determining the illumination coefficient distribution information of the first preview image, the method further includes: determining multiple sub-regions in the subject edge region according to a preset region division rule; each sub-region corresponds to a partition type; and determining the illumination coefficient of the subject edge region based on the illumination coefficient corresponding to the partition type to which each sub-region belongs.

[0039] For example, a sub-region can be the main sub-image region mentioned in the following embodiments, and the partition type corresponds to the first type partition, the second type partition, the third type partition, and the fourth type partition mentioned in the following embodiments. The illumination coefficient can be the illumination coefficient variation curve mentioned in the following embodiments, and the illumination coefficient variation curves corresponding to sub-regions of different partition types are different.

[0040] In this way, the edge region of the subject is first divided into zones based on the illumination conditions of different sub-regions; then, the illumination coefficient distribution information is automatically generated based on the illumination coefficient change curve corresponding to the zone type to which the subject sub-image region belongs, which can improve the accuracy of the determination of the illumination coefficient distribution information.

[0041] According to the first aspect, or any implementation of the first aspect above, in this method, performing image fusion processing based on the first preview image and the light-weighted rendering image to obtain the second preview image includes: performing color gamut adjustment processing on the first preview image to obtain the image to be lit; and performing image fusion processing based on the image to be lit and the light-weighted rendering image to obtain the second preview image.

[0042] Therefore, considering that if the initial image brightness is relatively high, the addition of lighting effects may lead to overexposure, a process is added to determine whether the initial image brightness meets preset constraints. If the initial image brightness meets the preset constraints (e.g., the image brightness of the background image area is less than the first threshold, and the image brightness of the main image area is less than the second threshold), then the image color gamut adjustment process can be skipped. Only when the initial image brightness does not meet the preset constraints will the image color gamut adjustment process be triggered to avoid overexposure of the target lit image.

[0043] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining multiple first lighting templates; the multiple first lighting templates include a real-shot lighting distribution template and a simulated lighting distribution template; performing lighting color conversion processing based on any one of the first lighting templates to obtain a second lighting template; and generating a lighting template set based on the multiple first lighting templates and the multiple second lighting templates.

[0044] For example, the first lighting template may be the basic lighting distribution template mentioned in the following embodiments, and the second lighting template may be the extended lighting distribution template mentioned in the following embodiments.

[0045] In this way, by converting the light color based on the available basic lighting modules, an expanded lighting template can be obtained, thereby ensuring the diversity of the lighting template set and meeting the diverse lighting needs of different users for background lighting effects.

[0046] According to the first aspect, or any implementation of the first aspect above, the method further includes: displaying a fourth preview interface of the target application; the fourth preview interface includes a seventh control; in response to a ninth operation on the seventh control, displaying a fifth preview interface of the target application; the fifth preview interface includes a sixth preview image, the sixth preview image being obtained by illuminating the target object in the fifth preview image based on a simulated light source, and the fifth preview image being a frame preview image captured by an electronic device.

[0047] For example, the fourth preview interface can be the same as the first preview interface, the seventh control can be the same as the first control, and the fifth preview interface can be the same as the second preview interface. Specifically, the first preview interface of the target application is displayed; in response to a first operation on the first control, the target lighting type is determined; the target lighting type includes either subject lighting or background lighting; the second preview interface of the target application is displayed; wherein, if the target lighting type is background lighting, the second preview interface includes a second preview image, and if the target lighting type is subject lighting, the second preview interface includes a sixth preview image; the simulated light source can be the simulated lighting source mentioned in the following embodiments, where a simulated light source refers to a light source created in a digital environment to simulate the characteristics of a light source in the real world and to simulate the lighting effect of a real light source.

[0048] In this way, by providing users with two image processing functions, namely subject lighting and background lighting, users can decide which image lighting function to trigger based on their actual needs, thus meeting their different lighting requirements in different scenarios.

[0049] Specifically, the process of lighting the target object in the image first involves automatically simulating and generating illumination distribution information for the target object based on a simulated light source. Then, this illumination distribution information is used to light the target object in the fifth preview image, thereby improving the realism of the image lighting effect. Optionally, to further enhance the realism of the image lighting effect, a three-dimensional model of the target object can be constructed first. Based on the simulated light source and the three-dimensional model, the spatial illumination distribution information of the target object can be simulated and generated. Then, this spatial illumination distribution information is used to light the target object in the fifth preview image.

[0050] According to the first aspect, or any implementation of the first aspect above, in this method, before displaying the fifth preview interface of the target application, the method further includes: displaying a third settings interface, and in response to a tenth operation performed on the third settings interface, determining fourth parameter information of the simulated light source; and performing lighting processing on the target object in the fifth preview image according to the fourth parameter information to obtain a sixth preview image.

[0051] For example, the third setting interface includes at least one parameter setting control, each parameter setting control being used to set lighting parameters; the third setting interface may be the lighting parameter setting interface mentioned in the embodiments below; the parameter setting control may be a control used to set the value of a certain lighting parameter; the fourth parameter information includes the parameter values ​​corresponding to multiple lighting parameters respectively.

[0052] In this way, after detecting the trigger operation of the lighting control, a lighting parameter setting interface is provided to the user. The user can set the value of the image lighting parameter according to the actual needs, so as to meet the user's personalized needs for background lighting effect.

[0053] According to the first aspect, or any implementation of the first aspect above, in this method, the lighting process of the target face in the fifth preview image based on the fourth parameter information to obtain the sixth preview image may include: extracting key points from the fifth preview image to obtain key point distribution information of the target face in the fifth preview image; performing three-dimensional reconstruction based on the key point distribution information to obtain a three-dimensional model of the target face; performing simulated lighting processing on the three-dimensional model based on the fourth parameter information to obtain target illumination distribution information corresponding to the three-dimensional model; and performing image fusion processing based on the fifth preview image and the target illumination distribution information to obtain the sixth preview image.

[0054] For example, the fifth preview image may be the initial image mentioned in the following embodiments, and the sixth preview image may be the target lighting image mentioned in the following embodiments. The key point distribution information may be the key point spatial distribution information mentioned in the following embodiments, which includes the spatial position information of the key points of the target object; the target illumination distribution information may be the target illumination spatial distribution information mentioned in the following embodiments, which reflects the illumination distribution in three-dimensional space, and therefore can also be called a three-dimensional spatial distribution map of illumination intensity; the image fusion process may mainly include first projecting each pixel in the target illumination distribution information onto a two-dimensional plane distribution map, then weighting and summing the pixel values ​​of each pixel in the two-dimensional plane distribution map with the pixel values ​​of the corresponding pixels in the first preview image to obtain the fused pixel value; and then generating the target lighting image based on the fused pixel values ​​of each pixel.

[0055] Thus, in the process of lighting a target object in an image, if the target face to be lit is to be the subject of the lighting effect, the process begins by extracting facial key points from the image. Then, a 3D facial model is constructed based on these key points. Next, based on the 3D facial model and the light source parameters of the simulated lighting source, the target face's spatial illumination distribution information (including at least the spatial illumination distribution information corresponding to specular reflection and diffuse reflection) is generated. Finally, this spatial illumination distribution information is fused with the initial image to obtain the target lit image. This process simulates the lighting direction and intensity corresponding to each unit surface in the 3D facial model, making the lighting effect added to the initial image closer to real lighting, thereby ensuring the authenticity of the target lit image displayed to the user.

[0056] According to the first aspect, or any implementation of the first aspect above, in this method, the three-dimensional reconstruction process based on the key point distribution information is performed to obtain a three-dimensional model of the target face, which may include: selecting a first point and a second point from multiple key points in the key point distribution information; using the neighborhood growth method, starting from the first point and ending at the second point, sequentially determining the current growth point from multiple key points; establishing the connection relationship between the current growth point and the neighborhood growth point of the current growth point to obtain a three-dimensional model of the target face.

[0057] For example, the first point may be the starting growth point mentioned in the embodiments below, and the second point may be the ending growth point mentioned in the embodiments below.

[0058] In this way, since the 3D face model is obtained by establishing the connection relationship between the key points of the face using the neighborhood growth method, the 3D face model can include multiple unit surfaces. Each unit surface is used as the smallest computing unit to calculate the reflected light intensity corresponding to each unit surface in the 3D face model. Then, the pixel value of each pixel point on the unit surface is determined. Then, the target illumination spatial distribution information is generated based on the pixel value. This not only ensures the accuracy of the calculation of reflected light intensity, but also improves the generation efficiency of illumination spatial distribution information.

[0059] According to the first aspect, or any implementation of the first aspect above, in this method, simulating lighting processing on the three-dimensional model based on the fourth parameter information to obtain the target illumination distribution information corresponding to the three-dimensional model may include: determining the target parameter information of each unit surface in the three-dimensional model based on the fourth parameter information; calculating specular reflection illumination based on the target parameter information to obtain the first illumination distribution information; calculating diffuse reflection illumination based on the target parameter information to obtain the second illumination distribution information; and determining the first illumination distribution information and the second illumination distribution information as the target illumination distribution information.

[0060] For example, the target parameter information can be the target illumination parameter information mentioned in the following embodiments; wherein, the fourth parameter information refers to the relevant parameter information of the simulated light source itself, and the target parameter information refers to the relevant parameter information presented after the simulated light source illuminates the object surface. The first illumination distribution information can be the first illumination spatial distribution information mentioned in the following embodiments, and the second illumination distribution information can be the second illumination spatial distribution information mentioned in the following embodiments.

[0061] Thus, considering that the initial image (such as the fifth preview image) already contains the luminous effects brought about by ambient light and self-emission, the four types of lighting conditions—ambient light, self-emission, specular reflection, and diffuse reflection—are distinguished. Only the spatial distribution information of the lighting corresponding to specular reflection and the spatial distribution information of the lighting corresponding to diffuse reflection are generated. This ensures that the pixel value of each pixel in the target lighting spatial distribution information is only related to the intensity of specular reflection and diffuse reflection, and is independent of the intensity of the lighting produced by ambient light and self-emission. This avoids the problem of oversaturation of the lighting in the target lighting image due to the repeated superposition of the luminous effects brought about by ambient light and self-emission.

[0062] According to the first aspect, or any implementation of the first aspect above, in this method, calculating the specular reflection illumination based on the target parameter information to obtain the first illumination distribution information may include: calculating the specular reflection illumination based on the target parameter information to obtain the first initial distribution information; and smoothing the first initial distribution information to obtain the first illumination distribution information.

[0063] Correspondingly, diffuse reflection illumination is calculated based on the target parameter information to obtain the second illumination distribution information, including: calculating diffuse reflection illumination based on the target parameter information to obtain the second initial distribution information; and smoothing the second initial distribution information to obtain the second illumination distribution information.

[0064] For example, the first initial distribution information may be the initial illumination spatial distribution information corresponding to specular reflection mentioned in the following embodiments, and the second initial distribution information may be the initial illumination spatial distribution information corresponding to diffuse reflection mentioned in the following embodiments. The first initial distribution information is used to characterize the specular reflection illumination distribution of each unit surface in the three-dimensional model of the target face, and the second initial distribution information is used to characterize the diffuse reflection illumination distribution of each unit surface in the three-dimensional model of the target face.

[0065] Thus, considering that the three-dimensional model is composed of multiple triangular facets, due to the discreteness of the triangular facets, the lighting effect shown by the initial lighting spatial distribution information is not smooth and uniform, but rather appears as patchy spots. Therefore, in order to reduce the transitional differences in lighting intensity between adjacent triangular facets, thereby ensuring that the lighting effect shown by the lighting spatial distribution information is smoother.

[0066] According to the first aspect, or any implementation of the first aspect above, in this method, the target illumination distribution information includes first illumination distribution information and second illumination distribution information; image fusion processing based on the fifth preview image and the target illumination distribution information to obtain a sixth preview image may include: determining the first pixel value of the target pixel based on the first illumination distribution information; and determining the second pixel value of the target pixel based on the second illumination distribution information; the target pixel is any pixel on the three-dimensional model of the target face; performing a weighted summation of the original pixel value, the first pixel value, and the second pixel value of the target pixel to obtain a fused pixel value; wherein the original pixel value is used to characterize the pixel color of the fifth preview image; generating the sixth preview image based on the original pixel values ​​of each non-target pixel in the fifth preview image and the fused pixel values ​​of each target pixel; the non-target pixels are pixels in the fifth preview image that have not been illuminated.

[0067] For example, each first pixel value is used to characterize the specular reflection color of any pixel in a triangular facet, and each second pixel value is used to characterize the diffuse reflection color of any pixel in a triangular facet.

[0068] In this way, since the difference in reflected light intensity among pixels in a certain triangular facet of the target facet is relatively small, the reflected light color of each pixel in the certain triangular facet can be regarded as the same. Consequently, the first pixel value or the second pixel value of each pixel in the certain triangular facet can be regarded as the same, thereby improving the efficiency of determining the fused pixel value of each pixel in the three-dimensional facet.

[0069] According to the first aspect, or any implementation of the first aspect above, in this method, the target face is the target face, and the fifth preview image also includes the target human body corresponding to the target face; after performing simulated lighting processing on the three-dimensional model according to the fourth parameter information to obtain the target illumination distribution information corresponding to the three-dimensional model, the method further includes: predicting the human body illumination distribution according to the target illumination distribution information to obtain the human body illumination distribution information of the target human body.

[0070] Correspondingly, the sixth preview image is obtained by performing image fusion processing based on the fifth preview image and the target illumination distribution information. This may include: performing human image synthesis processing based on the target illumination distribution information and the human body illumination distribution information to obtain overall illumination distribution information; and performing image fusion processing on the fifth preview image and the overall illumination distribution information to obtain the sixth preview image.

[0071] For example, the human body illumination distribution information may include the third illumination plane distribution information and the fourth illumination plane distribution information of the target human body mentioned in the following embodiments; the overall illumination distribution information includes the first overall illumination distribution information and the second overall illumination distribution information mentioned in the following embodiments.

[0072] In this way, for the case where the initial image includes the target human figure, the target face and the target body are segmented into two image regions. First, the facial illumination distribution information of the target face (i.e., the target illumination distribution information) is determined. Then, based on the target illumination distribution information, the body illumination distribution information of the target body is predicted. Then, based on the combination of the facial illumination distribution information and the body illumination distribution information, the overall illumination distribution information is obtained. Then, based on the fifth preview image and the overall illumination distribution information, the fused pixel value of each pixel in the target human figure is determined, and thus the sixth preview image (i.e., the target lighting image) is obtained, thereby ensuring the coordination of the lighting effect between the target face and the target body in the sixth preview image.

[0073] According to the first aspect, or any implementation of the first aspect above, in this method, the target face is the target face, and the fifth preview image also includes a background shooting object with a certain positional relationship to the target face; after performing simulated lighting processing on the three-dimensional model according to the fourth parameter information to obtain the target illumination distribution information corresponding to the three-dimensional model, the method further includes: predicting the background illumination distribution according to the target illumination distribution information and positional relationship to obtain the background illumination distribution information of the background shooting object.

[0074] Correspondingly, the image fusion processing based on the fifth preview image and the target illumination distribution information to obtain the sixth preview image may include: performing a synthesis processing based on the target illumination distribution information and the background illumination distribution information to obtain panoramic illumination distribution information; and performing image fusion processing on the fifth preview image and the panoramic illumination distribution information to obtain the sixth preview image.

[0075] For example, the background illumination distribution information may include the fifth illumination plane distribution information and the sixth illumination plane distribution information of the background shooting object mentioned in the following embodiments; the panoramic illumination distribution information includes the first panoramic illumination distribution information and the second panoramic illumination distribution information mentioned in the following embodiments.

[0076] In this way, for an initial image that includes a target face and a shooting background, the target face and the shooting background are divided into two image regions. First, the facial illumination distribution information of the target face (i.e., target illumination distribution information) is determined. Then, based on the target illumination distribution information, the background illumination distribution information of the shooting background is predicted. Then, the panoramic illumination distribution information is obtained by combining the facial illumination distribution information and the background illumination distribution information. Finally, based on the fifth preview image and the panoramic illumination distribution information, the fusion pixel value of each pixel in the face and the background is determined, thus obtaining the sixth preview image (i.e., target lighting image), thereby ensuring the coordination of the lighting effect between the target face and the shooting background in the sixth preview image.

[0077] In this context, any one of the first preview image, third preview image, first captured image, third captured image, first image, and fifth preview image can be the initial image mentioned in the embodiments below. If the initial image is the first preview image, then the target lighting image corresponding to the initial image is the second preview image mentioned above; if the initial image is the first captured image, then the target lighting image corresponding to the initial image is the second captured image mentioned above; if the initial image is the first image, then the target lighting image corresponding to the initial image is the second image mentioned above; if the initial image is the third preview image, then the target lighting image corresponding to the initial image is the fourth preview image mentioned above; if the initial image is the third captured image, then the target lighting image corresponding to the initial image is the fourth captured image mentioned above; if the initial image is the fifth preview image, then the target lighting image corresponding to the initial image is the sixth preview image mentioned above. Furthermore, the lighting processing process for any one of the third preview image, first captured image, third captured image, first image, and fifth preview image can be referred to the lighting processing process for the first preview image, and will not be repeated here.

[0078] Secondly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, the electronic device performs the image processing method as shown in the first aspect and any one of the first aspects.

[0079] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0080] For example, the electronic device may be a terminal device or a chip within a terminal device. The electronic device may include an input unit and a processing unit. When the electronic device is a terminal device, the processing unit may be a processor, and the input unit may be a communication interface; the terminal device may also include a memory for storing computer program code, which, when executed by the processor, causes the terminal device to perform any of the image processing methods described in the first aspect.

[0081] When the electronic device is a chip within a terminal device, the processing unit can be an internal processing unit of the chip, and the input unit can be an output interface, pin, or circuit, etc.; the chip may also include a memory, which can be an internal memory of the chip (e.g., registers, cache, etc.) or an external memory (e.g., read-only memory, random access memory, etc.); the memory is used to store computer program code, and when the processor executes the computer program code stored in the memory, the chip performs any of the image processing methods in the first aspect.

[0082] Thirdly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the image processing method of the first aspect and any one thereof.

[0083] The third aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0084] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when run, causes a computer to perform an image processing method as described in the first aspect or any one of the first aspects.

[0085] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0086] Fifthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes an image processing method as described in the first aspect or any one thereof, to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

[0087] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0088] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0089] Figure 1 This is an illustrative diagram of a portrait photography scene;

[0090] Figure 2 This is a schematic diagram illustrating the processing flow provided in an embodiment of this application, which is an example of such an application.

[0091] Figure 3 A schematic diagram of the hardware structure of an electronic device as an example;

[0092] Figure 4 A schematic diagram of the software architecture of an electronic device as an example;

[0093] Figures 5 to 9 This is a schematic diagram illustrating the application scenario of the electronic device provided in the embodiments of this application;

[0094] Figure 10 This is a schematic diagram illustrating the application scenario of the electronic device provided in the embodiments of this application;

[0095] Figure 11A schematic diagram of module interaction of an electronic device provided in an embodiment of this application;

[0096] Figure 12 A schematic diagram illustrating the construction process of the lighting template set provided in the embodiments of this application;

[0097] Figure 13a A schematic diagram illustrating the determination of the inner edge contour of the subject under positive and backlighting conditions, as provided in an embodiment of this application.

[0098] Figure 13b A schematic diagram illustrating the determination of the projected edge contour of the subject under side-backlighting, as provided in an embodiment of this application.

[0099] Figure 14 Schematic diagram of illumination coefficient variation curves corresponding to the four types of partition distribution provided in the embodiments of this application;

[0100] Figure 15a A schematic diagram illustrating the partitioning type of the subject sub-image region under positive and backlighting conditions provided in an embodiment of this application;

[0101] Figure 15b A schematic diagram illustrating the partitioning type of the subject sub-image region under side backlighting provided in an embodiment of this application;

[0102] Figure 15c A schematic diagram illustrating the partitioning type of a sub-image region within a magnified image area under side backlighting, provided in an embodiment of this application.

[0103] Figure 16a This is a schematic diagram illustrating the image processing flow under positive and backlighting conditions provided in an embodiment of this application, which is an example of such an embodiment.

[0104] Figure 16b This is a schematic diagram illustrating the image processing flow under side backlighting provided in an embodiment of this application, which is an example of such a process.

[0105] Figure 17 This is a schematic diagram illustrating the process of selecting the target shooting object provided in an embodiment of this application;

[0106] Figures 18a to 18d A schematic diagram of the framework of an electronic device provided in an embodiment of this application;

[0107] Figures 19a to 19c This is a schematic diagram illustrating the process of setting the image lighting type according to an embodiment of this application.

[0108] Figure 20 A schematic diagram of module interaction of an electronic device provided in an embodiment of this application;

[0109] Figure 21 A schematic diagram illustrating the construction process of the 3D face model provided in this application embodiment;

[0110] Figure 22 A schematic diagram illustrating the process of determining the target lighting parameters for a unit surface provided in an embodiment of this application;

[0111] Figure 23 This is a schematic diagram illustrating the processing flow provided in an embodiment of this application, which is an example of such an application.

[0112] Figure 24 A schematic diagram of the processing flow of the image processing method provided in the embodiments of this application;

[0113] Figure 25 This is a schematic diagram of the processing flow of the image processing method provided in the embodiments of this application. Detailed Implementation

[0114] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0115] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0116] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0117] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0118] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0119] To facilitate understanding, the following explanations will be provided for some of the technical terms, related terms and concepts involved in the embodiments provided in this application.

[0120] Light source direction: refers to the direction in which the light source shines. For backlit shooting scenes, the light source direction can be divided into direct backlighting and side backlighting, which are used to describe the relative position of the light source and the subject being photographed.

[0121] Backlighting: This refers to a light source located directly behind or nearly behind the subject, where the light shines on the back of the subject. In this case, the light source illuminates the back of the subject while also illuminating the outer edge of the subject.

[0122] Side-backlighting: This refers to a light source that is positioned behind the subject and to one side of it. In other words, the light comes from one side of the subject, illuminating it from one side while creating a shadow on the other side.

[0123] A lighting template set refers to a collection of pre-built lighting distribution templates. It can include a basic lighting template subset and an extended lighting template subset. The basic lighting template subset includes multiple basic lighting distribution templates, and the extended lighting template subset includes multiple extended lighting distribution templates. The extended lighting distribution templates are obtained by mapping the lighting colors of the basic lighting distribution templates.

[0124] Illumination distribution template: refers to the illumination intensity distribution information formed when a light source with certain light source parameters illuminates a specified background.

[0125] Illumination coefficient distribution information: refers to the distribution information of the proportional coefficient of the light intensity to be added to each pixel in the image.

[0126] Light-weighted rendering map: This refers to the light intensity distribution map obtained by element-wise difference between the light distribution template and the light coefficient distribution information. This process can be understood as a masking operation, that is, using the light coefficient distribution information (which can be called a mask image) to control the light intensity of the main area of ​​the template (such as a photo) to become 0 or change according to a specified change law (related to the light intensity coefficient change curve), while the light intensity of the background area of ​​the template remains unchanged (i.e., the light coefficient is equal to 1). Here, the main area of ​​the template refers to the area corresponding to the target subject in the image to be lit, and the background area of ​​the template refers to the area corresponding to the background area of ​​the image to be lit.

[0127] Background lighting image: refers to the fused image obtained by image fusion processing of the light weighted rendering image and the image to be lit; the image fusion processing mainly includes weighted summation of the pixel values ​​of each pixel in the light weighted rendering image and the corresponding pixel values ​​in the image to be lit to obtain the fused pixel value; then, the background lighting image is generated based on the fused pixel values ​​of each pixel.

[0128] Facial landmarks refer to key feature points extracted from a facial image region, which may include feature points of prominent parts such as eyes, nose, mouth, eyebrows, and facial contours.

[0129] Spatial distribution information of key points of the target face: used to characterize the positional distribution of key points of the face in three-dimensional space, which may include the three-dimensional coordinate information of the key points of the face.

[0130] 3D Face Model: By traversing the spatial distribution information of facial key points, establishing connections between these key points, a 3D face model is obtained. The outer surface of the 3D face model can include multiple facial unit faces. A facial unit face can be a triangular face containing three facial key points, meaning each triangular face can be considered a unit face. Alternatively, a facial unit face can be a quadrilateral containing four facial key points, meaning any quadrilateral can be considered a facial unit face.

[0131] Unit face normal vector: refers to the normal vector of a certain face unit face in a 3D face model. The direction of the normal vector is perpendicular to the triangle face.

[0132] High-reflection: When light shines on the surface of an object, the light is concentrated and reflected in a specific direction. This phenomenon is called high-reflection.

[0133] Diffuse reflection: When light shines on the surface of an object, the light is scattered randomly in various directions. This phenomenon is called diffuse reflection.

[0134] Highlight reflection intensity: refers to the light intensity in the direction of highlight reflection on a unit face surface in a 3D face model.

[0135] Diffuse light intensity refers to the light intensity produced by diffuse reflection from a specific face unit surface in a 3D face model. Furthermore, considering that the reflection direction of diffuse reflection from each face unit surface is relatively dispersed, the diffuse light intensity from adjacent face unit surfaces may be projected onto a particular face unit surface. Therefore, the diffuse light intensity of a specific face unit surface can be determined by a combination of the diffuse light intensity produced by the face unit surface itself and the diffuse light intensity produced by adjacent face unit surfaces.

[0136] Pixel values ​​in an image: These are the RGB values ​​used to represent a specific pixel. Various colors are obtained by varying the red, green, and blue color channels and their combinations. RGB represents the colors of these three channels, and the RGB value is an integer representing the values ​​of these three channels for a given pixel. Typically, there are 256 possible RGB values, ranging from 0, 1, 2... up to 255. For example, the RGB value of a pixel could be (122, 255, 0).

[0137] The specific implementation process of some embodiments provided in this application will be described in detail below.

[0138] Currently, considering that taking photos in low-light conditions may result in unsatisfactory image quality, lighting processing is necessary to ensure the best possible image display. (See also...) Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of an image processing method. For example... Figure 1 As shown, in portrait photography, to highlight the subject's contours and achieve a backlit effect, a professional light source needs to be pre-positioned at the shooting location, either directly behind or to one side of the subject. The subject is then photographed under this light, resulting in a portrait image with background lighting. However, limitations in the selection and placement of the light source can lead to a lack of diversity in the background lighting effects of the final portrait image. Furthermore, constraints imposed by the shooting scene and light source can make it difficult to obtain images with the desired background lighting effect.

[0139] To address the aforementioned problems, this application provides an image processing method applied to electronic devices (such as mobile phones). This method can be used in everyday shooting scenarios, automatically performing background lighting processing on an initial image to obtain a background-lit image that meets expectations, without requiring the deployment of a professional shooting light source; or it can perform background lighting processing on sample images to obtain sample images with diverse background lighting effects. See also... Figure 2 , Figure 2 This is a schematic diagram of the processing flow provided in an embodiment of this application. Figure 2As shown, an initial image to be processed is acquired, which includes at least one target subject. The initial image undergoes color gamut adjustment to obtain a lighting image; and subject segmentation is performed on the initial image to determine the contour position information of the target subject. Furthermore, a target lighting distribution template is determined from a preset set of lighting distribution templates based on the light source parameter information of the simulated light source. Then, based on the contour position information of the target subject and the light source direction, the subject edge region and background region in the initial image are determined; lighting coefficient distribution information is generated based on the lighting coefficient variation curve of the background region and the lighting coefficient variation curves corresponding to each subject sub-image region in the subject edge region. A lighting weighted rendering map is generated based on the target lighting distribution template and the lighting coefficient distribution information; and image fusion processing is performed on the lighting image and the lighting weighted rendering map to obtain the target lighting image.

[0140] In the image processing method provided in this application embodiment, for the background lighting process, firstly, a target lighting distribution template corresponding to the simulated light source parameter information is selected from a preset lighting distribution template set; then, the contour information of the target subject is determined, and the subject edge region in the initial image is determined by combining the contour information of the target subject; then, the subject edge region is partitioned by combining the lighting conditions of different sub-regions in the subject edge region, and lighting coefficient distribution information is automatically generated based on the lighting coefficient change curve corresponding to the subject sub-image region; next, a lighting weighted rendering map is generated based on the target lighting distribution template and the lighting coefficient distribution information, and image fusion processing is performed based on the first preview image and the lighting weighted rendering map to obtain the target lighting image. This not only adds lighting effects to the background area, but also adds lighting effects to the critical area between the background area and the target subject (which can be called the subject edge region), thereby improving the overall visual coordination of the image lighting effect.

[0141] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0142] The technical solutions provided in this application can be applied to electronic devices with shooting capabilities. In some embodiments, the electronic device may be a mobile phone, tablet computer, handheld computer, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device, etc. The embodiments of this application do not impose any special limitations on the specific type of electronic device.

[0143] First, a detailed illustrative explanation of the hardware structure of the electronic device is provided.

[0144] For example, taking a mobile phone as an electronic device, Figure 3 A schematic diagram of the hardware structure of the electronic device is shown. It should be understood that... Figure 3 The electronic device 100 shown is merely an example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 3 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0145] like Figure 3As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0146] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0147] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0148] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0149] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0150] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0151] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0152] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0153] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0154] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0155] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0156] In this embodiment of the application, the display screen 194 can be used to display pages required by the electronic device (e.g., a camera interface, etc.), and display images captured by any one or more cameras 193 in the interface.

[0157] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0158] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0159] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0160] In this embodiment, the type of camera 193 can be distinguished based on hardware configuration and physical location. For example, a camera located on the side of the electronic device's display screen 194 can be called a front-facing camera, and a camera located on the back cover of the electronic device can be called a rear-facing camera. As another example, a camera with a short focal length and a wide field of view can be called a wide-angle camera, while a camera with a long focal length and a narrow field of view can be called a standard camera. Here, focal length and field of view are relative concepts without specific parameter limitations. Therefore, wide-angle cameras and standard cameras are also relative concepts, and can be specifically distinguished based on physical parameters such as focal length and field of view.

[0161] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 is in frequency selection mode, the DSP is used to perform Fourier transforms on the frequency energy, etc.

[0162] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0163] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0164] Next, a detailed illustrative description of the software architecture of electronic device 100 will be provided.

[0165] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software architecture of electronic device 100. Figure 4 A schematic diagram of the software architecture of an electronic device 100 according to an embodiment of this application is shown. It is understood that... Figure 4 The layers and components contained in each layer of the illustrated software architecture are merely an example and do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components, or combine some components, or split some components, or arrange the components differently; this application does not impose any limitations.

[0166] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android Runtime and system libraries, the hardware abstraction layer (HAL layer), and the kernel layer.

[0167] The application layer can include a series of application packages. For example... Figure 4 As shown, the application package can include applications (apps) such as camera, gallery, WLAN, and Bluetooth. The application package can also include applications such as calling, calendar, maps, navigation, music, video, and SMS.

[0168] In this embodiment of the application, an application with a photo-taking function (such as a camera application) responds to a user's trigger operation by calling the camera HAL in the hardware abstraction layer through the camera service in the application framework layer to control the camera driver in the kernel layer, thereby controlling the camera to take photos or videos as needed by the user.

[0169] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0170] like Figure 4 As shown, the application framework layer may include a window manager, content provider, resource manager, view system, notification manager, etc. The application framework layer may also include camera services and image processing services.

[0171] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0172] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0173] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0174] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0175] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0176] In this embodiment, the camera service controls the camera through the camera HAL in the hardware abstraction layer and the camera driver in the kernel layer to acquire the initial image captured by the camera in real time. When the application processor performs image lighting processing, the image processing service performs lighting processing on the initial image to obtain the target lit image.

[0177] For example, in the case of background lighting, the image processing service may specifically include an image preprocessing module, a lighting template determination module, a lighting coefficient determination module, and an image fusion module. The image preprocessing module performs color gamut adjustment on the initial image to obtain the image to be lit; and performs subject segmentation on the initial image to determine the contour position information of the target subject. The lighting template determination module determines the target lighting distribution template from a preset set of lighting distribution templates based on the light source parameter information of the simulated light source. The lighting coefficient determination module determines the subject edge region in the initial image based on the contour position information of the target subject, and generates lighting coefficient distribution information based on the lighting coefficient variation curves corresponding to each subject sub-image region within the subject edge region. The image fusion module generates a weighted lighting rendering map based on the target lighting distribution template and the lighting coefficient distribution information; and performs image fusion processing on the image to be lit and the weighted lighting rendering map to obtain the target lit image.

[0178] For example, in the case of subject lighting in an image, the image processing service may specifically include a key point extraction module, a stereo model construction module, an illumination distribution determination module, and an image fusion module. The key point extraction module extracts key points from the target object (e.g., a face) in the initial image to obtain the spatial distribution information of the key points. The stereo model construction module performs 3D reconstruction based on the spatial distribution information of the key points to obtain a stereo model of the target object (e.g., a face stereo model). The illumination distribution determination module simulates lighting on the stereo model of the target object to obtain the spatial distribution information of the illumination of the target object. The image fusion module performs image fusion based on the initial image and the spatial distribution information of the illumination of the target object to obtain the target lighting image.

[0179] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0180] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0181] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0182] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0183] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0184] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0185] 3D graphics processing libraries are used to implement 3D graphics drawing, image rendering, compositing, and layer processing. 2D graphics engines are drawing engines for 2D graphics.

[0186] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and the hardware circuitry, its purpose being to abstract the hardware. It hides the platform-specific hardware interface details, providing the operating system with a virtual hardware platform that is hardware-independent and portable across multiple platforms. The HAL provides a standard interface that exposes device hardware functionality to the higher-level Java API framework (i.e., the framework layer). The HAL contains multiple library modules, each implementing an interface for a specific type of hardware component, such as: audio HAL (audio hardware abstraction module), Bluetooth HAL (Bluetooth hardware abstraction module), camera HAL (camera hardware abstraction module), and sensors HAL (sensor service).

[0187] In this embodiment, the camera HAL primarily serves as a bridge between the upper and lower layers. It provides its own methods (or functions or APIs) to the camera service through the HIDL interface of the HAL layer, enabling the camera service to communicate with the underlying driver (so that instructions from the camera service can be transmitted to the camera module, causing the camera module to operate according to the instructions). In this way, the application can control the camera driver by calling the camera HAL through the camera service, thereby controlling the camera to take pictures. When the image lighting processing is performed by the digital signal processor, the camera HAL also receives the initial image from the application framework layer and sends it to the digital signal processor to trigger the digital signal processor to perform lighting processing on the initial image to obtain the target lit image.

[0188] Correspondingly, for the case where the image lighting type is background lighting, the digital signal processor can include an image preprocessing module, a lighting template determination module, a lighting coefficient determination module, and an image fusion module. The image preprocessing module performs color gamut adjustment on the initial image to obtain the image to be lit; and performs subject segmentation on the initial image to determine the contour position information of the target subject. The lighting template determination module determines the target lighting distribution template from a preset set of lighting distribution templates based on light source parameter information. The lighting coefficient determination module determines the subject edge region in the initial image based on the contour position information of the target subject, and generates lighting coefficient distribution information based on the lighting coefficient variation curves corresponding to each subject sub-image region within the subject edge region. The image fusion module generates a weighted lighting rendering map based on the target lighting distribution template and the lighting coefficient distribution information; and performs image fusion processing on the image to be lit and the weighted lighting rendering map to obtain the target lit image.

[0189] In addition, for cases where the image lighting type is subject lighting, the digital signal processor can include a key point extraction module, a stereo model construction module, an illumination distribution determination module, and an image fusion module. The key point extraction module extracts key points from the target object (e.g., a face) in the initial image to obtain the spatial distribution information of the key points. The stereo model construction module performs 3D reconstruction based on the key point spatial distribution information to obtain a stereo model of the target object (e.g., a face stereo model). The illumination distribution determination module simulates lighting on the stereo model of the target object to obtain the spatial distribution information of the illumination of the target object. The image fusion module performs image fusion based on the initial image and the spatial distribution information of the illumination of the target object to obtain the target lighting image.

[0190] The kernel layer is the layer between hardware and software. It includes at least display drivers, camera drivers, Bluetooth drivers, and sensor drivers. The hardware includes at least a processor, display screen, camera, Bluetooth module, and sensors.

[0191] The display driver drives the screen in an electronic device to display information. The camera driver drives the camera in an electronic device to take pictures.

[0192] The technical solutions provided in the embodiments of this application can all be implemented in electronic devices with the above-described hardware or software architecture.

[0193] It is understood that, in order to implement the image processing methods in the embodiments of this application, electronic devices include hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0194] To facilitate understanding, before providing a detailed description of the image processing method provided in the embodiments of this application, the timing of triggering the initial image lighting process involved in the embodiments of this application will be explained. Figures 5 to 9 This is an illustrative diagram illustrating an application scenario of an electronic device. In this example, the electronic device is a mobile phone, and the target application is a camera application. In one scenario, during the photo preview process, if a user clicks on the lighting control, the captured image is not stored; instead, the target lighting image corresponding to the captured image is directly stored in a designated location and viewed through the gallery application. For example, if a user click on the lighting control is detected before the user clicks it, the preview image to be displayed can be processed to obtain the target lighting image. Then, the target lighting image is displayed in the preview area of ​​the camera application's photo preview interface so that the user can determine whether the lighting effect of the target lighting image meets expectations. If the user confirms that the background lighting effect meets expectations, the user can click the photo control. If the user clicks on the photo control, the camera captures an image, processes the captured image to obtain the corresponding target lighting image, and then stores the target lighting image. In other words, for a photography scenario, if a user clicks on the lighting control, the preview image needs to be lit before the user clicks the shutter button, and the corresponding target lighting image needs to be displayed in the preview area. After the user clicks the shutter button, the captured image needs to be lit, and the corresponding target lighting image needs to be stored. Alternatively, a historical image can be selected from the gallery application as the image to be lit, triggering the lighting process on that historical image to obtain the target lighting image, which is then stored in a designated location and viewed through the gallery application.

[0195] In some example embodiments, such as Figure 5 As shown, for an initial image to be illuminated that is a real-time captured preview image, see [reference needed]. Figure 5In step (a), the phone's home screen is displayed, and the user clicks the camera app icon 501; the phone responds to this user action and displays as shown below. Figure 5 The photo preview interface shown in (b) displays image data of the subject captured by the camera of the electronic device in preview area 502. At this time, the user can click the first lighting control 503 in the photo preview interface. The phone responds to this click by acquiring the next frame of the preview image to be displayed, and also acquiring the light source parameter information of the simulated lighting source. This light source parameter information can be automatically determined by the system. Based on this light source parameter information, lighting processing is performed on the preview image to be displayed to obtain the first lighting image (i.e., the target lighting image corresponding to the preview image). The preview image to be displayed is not... Figure 5 The preview image shown in (b) is displayed in preview area 502, but in the case of a detected user click on the first lighting control 503, the next frame of the preview image to be displayed. Next, the image shown is... Figure 5 As shown in (c) of the image preview interface, the preview area 502 displays the first lighting image. If the lighting effect of the first lighting image meets the user's expectations, the user can click the camera control 504. The phone responds to the click operation, captures the image through the camera, performs lighting processing on the captured image to obtain the second lighting image (i.e., the target lighting image corresponding to the captured image), and stores the second lighting image. The user can quickly view the second lighting image from the gallery by clicking the photo viewing control in the lower left corner of the image preview interface.

[0196] In addition, such as Figure 6 As shown, a lighting parameter setting interface can be provided to users in photography scenarios. (See [link]) Figure 6 In step (a), the phone's home screen is displayed, and the user clicks the camera app icon 501; the phone responds to this user action and displays as shown below. Figure 6 As shown in (b) of the image, the preview area 502 of the image preview interface displays the image data of the subject currently captured by the camera of the electronic device. At this time, the user can click the first lighting control 503 in the image preview interface; the mobile phone responds to the user's operation and displays as shown in the image preview interface. Figure 6 In the lighting parameter setting interface shown in (c), the user can customize the light source parameter information of the simulated lighting source by triggering the first parameter setting control 505. Then, the user clicks the first confirmation control 506, indicating that the lighting parameter setting is complete. The phone responds to this click by acquiring the next frame of the preview image to be displayed, as well as the light source parameter information of the simulated lighting source. Based on this light source parameter information, lighting processing is performed on the preview image to be displayed to obtain the first lit image. Then, the image is displayed as shown... Figure 6The photo preview interface shown in (d) displays a first lighting image in the preview area 502. If the user confirms that the lighting effect of the first lighting image meets expectations, the user can click the photo control 504. The mobile phone responds to the click operation, captures the image through the camera, and stores the second lighting image corresponding to the captured image.

[0197] Furthermore, such as Figure 7 As shown, for a scenario where the user triggers adjustments to the background lighting effect during the photo preview process, for example, taking the image relighting triggered by the first lighting control 503 as an example, see [link to example]. Figure 7 The photo preview interface shown in (a) displays a first illuminated image in preview area 502. If the user confirms that the lighting effect of the first illuminated image does not meet expectations, the user can click the first illuminated control 503. The phone responds to this click operation by displaying the following... Figure 7 In the lighting parameter setting interface shown in (b), the user can adjust the light source parameters of the simulated lighting source by triggering the first parameter setting control 505. Then, the user clicks the first confirmation control 506, indicating that the lighting parameters have been reset. The phone responds to this click by acquiring the next frame of the preview image to be displayed, as well as the adjusted light source parameters. Based on these adjusted light source parameters, the preview image to be displayed is processed with lighting to obtain the adjusted lighting image. Then, the image is displayed as shown... Figure 7 The photo preview interface shown in (c) displays the adjusted lighting image in the preview area 502. If the user confirms that the lighting effect of the adjusted lighting image meets expectations, the user can click the photo control 504. The mobile phone responds to the click operation, captures the image through the camera, performs lighting processing on the captured image based on the adjusted light source parameter information, and stores the target lighting image corresponding to the captured image.

[0198] Furthermore, such as Figure 8 As shown, the background lighting effect can be adjusted by triggering the newly added lighting effect input control. For example, taking the image relighting triggered by the human-computer interaction control 507 as an example, see [link to example]. Figure 8 The photo preview interface shown in (a) displays a first illuminated image in preview area 502. If the user confirms that the lighting effect of the first illuminated image does not meet expectations, the user can click the human-computer interaction control 507. The phone responds to this click operation by displaying the following... Figure 8The human-computer interaction interface 508 shown in (b) allows the user to input a description of the desired lighting effect by triggering the information input control 509. Then, the system automatically performs parameter matching based on the desired lighting effect description to obtain adjusted light source parameter information. Next, based on this adjusted light source parameter information, the next frame of the preview image to be displayed is lit to obtain an adjusted lighting image. Then, the image is displayed as shown in (b). Figure 8 The photo preview interface shown in (c) displays the adjusted lighting image in the preview area 502. If the user confirms that the lighting effect of the adjusted lighting image meets expectations, the user can click the photo control 504. The mobile phone responds to the click operation, captures the image through the camera, performs lighting processing on the captured image based on the adjusted light source parameter information, and stores the target lighting image corresponding to the captured image.

[0199] In this embodiment, a human-computer interaction control 507, different from the first lighting control 503, is added. Upon detecting a user's trigger operation on the human-computer interaction control 507, a human-computer interaction interface 508 is displayed. This human-computer interaction interface 508 can be the display interface of a voice assistant application. Based on this, upon detecting a user's trigger operation on the human-computer interaction control 507, the voice assistant application is launched, the human-computer interaction interface is displayed, and the user's input description of the desired lighting effect is listened to. Therefore, the adjusted light source parameter information can be obtained by matching parameter values ​​based on the user's input description of the desired lighting effect. For users who are unfamiliar with the correspondence between lighting parameter values ​​and lighting effects, they only need to describe the desired lighting effect without needing to pay attention to the specific values ​​of each lighting parameter.

[0200] Additionally, it is understandable that the first lighting control 503 can also trigger access to the human-computer interaction interface. For example, if the user clicks the first lighting control 503 again when the lighting image is displayed in the preview area, it means that the user is not satisfied with the lighting effect of the displayed lighting image and has a need to trigger the image to be relit. At this time, the human-computer interaction interface can be accessed directly, so that the user only needs to input the description of the desired lighting effect, thereby improving the adjustment effect of the light source parameter information.

[0201] In other example embodiments, such as Figure 9 As shown, for an initial image to be lit that is a historically captured image, see [reference needed]. Figure 9 In step (a), the phone's home screen is displayed, and the user clicks the gallery app icon 601; the phone responds to the user's action and displays as shown below. Figure 9In the photo display interface 602 shown in (b), after the user selects a photo as a gallery image, they can drag the gallery image towards the direction of the second lighting control 603 to indicate that the gallery image will be lit; the mobile phone responds to the user's operation and displays as shown in (b). Figure 9 In the lighting parameter setting interface shown in (c), the user sets the parameter values ​​of each lighting parameter by clicking the second parameter setting control 604, and then clicks the second confirmation control 605, indicating that the lighting parameter setting is complete. The phone responds to this click operation by acquiring the image from the user-selected gallery and the light source parameter information of the simulated lighting source. Based on this light source parameter information, the image from the gallery is processed to obtain the third lighting image. Next, the following is displayed... Figure 9 The image editing interface shown in (d) displays the third lighting image and the save control 606. If the user confirms that the lighting effect of the third lighting image meets expectations, the user can click the save control 606 to save the third lighting image. The user can also view the third lighting image in the all photos display interface. The user can also share, edit, delete, etc. of the third lighting image.

[0202] Additionally, if the user confirms that the lighting effect of the third lighting image does not meet expectations, they can trigger a relighting process for the image through a specified control; the process of determining the adjusted light source parameters can be referred to the above. Figure 7 and Figure 8 The specific implementation process shown will not be elaborated here.

[0203] Understandably, regarding the process of determining the light source parameters used for image lighting, after detecting a user's click on the lighting control (such as the first lighting control 503 or the second lighting control 603), the initial image can be directly lit based on the default light source parameters. This way, after the user clicks the lighting control, the target lit image is displayed directly without showing the lighting parameter settings interface. The default light source parameters can be user-preset parameters, system-recommended parameters, or parameters used by the user previously. For example, after detecting a user's click on the lighting control, the phone automatically obtains the default values ​​for the light source type, light source intensity, light source color, light source center position, and light source direction. Then, based on these default values, the image is lit to obtain the lit image. Optionally, after detecting a user's click operation on a lighting control (such as the first lighting control 503 or the second lighting control 603), a lighting parameter setting interface can also be provided to the user, allowing the user to personalize the lighting parameters according to actual needs; or the human-computer interaction interface can be triggered by a specified control, and parameter values ​​can be matched based on the user's input description of the desired lighting effect to obtain the light source parameter information of the simulated lighting source.

[0204] Furthermore, electronic devices can be not only mobile terminals such as smartphones, but also desktop PCs, which have image processing software installed. For example... Figure 10 As shown, for portrait photography scenarios, the initial image is first captured using a shooting device, without the need for a dedicated lighting source. Then, after acquiring the initial image, image processing software installed on a PC is used to process the image for lighting, resulting in a target-lit image. For background lighting, the image processing software may include an image preprocessing module, a lighting template determination module, a lighting coefficient determination module, and an image fusion module. For subject lighting, the digital signal processor may include a key point extraction module, a 3D model construction module, a lighting distribution determination module, and an image fusion module.

[0205] Next, the specific implementation process of the image processing method provided in this application embodiment will be described with reference to specific application scenarios. The following explanation uses a mobile phone as an example of an electronic device and a camera application as the target application. Other types of electronic devices and other applications with photography functions will not be described in this application embodiment.

[0206] Scene 1

[0207] In this scenario, taking an initial image containing a subject (such as a portrait) that requires enhanced background lighting as an example, and using this subject as the target subject, this scenario involves portrait photography using an electronic device. To achieve a backlit effect for the subject in the image, the background area of ​​the initial image is illuminated to obtain the target lit image. The light source direction can be either direct backlighting or side backlighting, such as... Figure 11 As shown, taking the light source direction as positive and backlighting directions as an example, the specific implementation process of generating the target lighting image corresponding to the initial image can include the following steps:

[0208] S701, the image preprocessing module performs color gamut adjustment on the initial image to obtain the image to be illuminated.

[0209] For example, color gamut adjustment processing may include at least one of brightness adjustment, color correction, color level adjustment, color balance, saturation adjustment, and contrast adjustment; see the specific implementation process of existing technologies, which will not be repeated here.

[0210] The initial image can be sent by the target application to the image preprocessing module for image preprocessing. The target application can be any application installed on the mobile phone, such as a camera application, a gallery application, or any other application with shooting function. Alternatively, the initial image can be sent by the image acquisition module of the target application to the image preprocessing module for image preprocessing. Correspondingly, the target application can also be a software application on a PC device.

[0211] Specifically, the target application sends the initial image to be processed to the image preprocessing module. After acquiring the initial image, the image preprocessing module performs a first preprocessing step, which may be color gamut adjustment. In one scenario, the initial image can be a preview image to be displayed on the camera application's preview interface. The phone detects the user's trigger operation on the camera application icon, launches the camera application, and captures the preview image through the camera. In another scenario, the initial image can be a captured image. The phone detects the user's trigger operation on the camera application's shutter control and captures the captured image through the camera. In yet another scenario, the initial image can be a historical image from a gallery application. The phone detects the user's trigger operation on the gallery application and determines the photo selected by the user from the gallery application interface as the initial image. In yet another scenario, the initial image can be a portrait image captured using professional photography equipment and transmitted to an electronic device (such as a PC). After acquiring the initial image, the target application sends it to the image preprocessing module and the image fusion module.

[0212] In addition, considering that if the initial image brightness is relatively high, the addition of lighting effects may lead to overexposure, a process is added to determine whether the initial image brightness meets preset constraints. If the initial image brightness meets the preset constraints (e.g., the image brightness of the background image area is less than the first threshold, and the image brightness of the main image area is less than the second threshold), then the image color gamut adjustment process can be skipped. Only when the initial image brightness does not meet the preset constraints will the image color gamut adjustment process be triggered to avoid overexposure of the target lighting image.

[0213] S702, the image preprocessing module performs subject segmentation processing on the initial image to determine the outline and position information of the target subject.

[0214] After acquiring the initial image to be processed, it is necessary not only to adjust the color gamut of the initial image, but also to determine the outline position information of the target subject in the initial image, and then determine the illumination distribution of the critical region between the target subject and the image background based on this position information. Specifically, after acquiring the initial image, the image preprocessing module performs a second preprocessing on the initial image. The second preprocessing can be subject segmentation, that is, segmenting the image region where the target subject is located from the initial image.

[0215] For example, the contour position information of the target subject can be obtained by segmenting the initial image using a pre-trained subject segmentation model. The subject segmentation model is obtained by iteratively updating the parameters of a neural network model based on a set of image samples. The subject segmentation model can learn how to extract the contour information of the target subject in the image, thereby determining the contour position information of the target subject in the initial image. Then, the image preprocessing module sends the contour position information of the target subject to the illumination coefficient determination module, and then, based on the contour position information of the target subject, determines the proportional coefficient distribution information of the light intensity to be added to each pixel in the image to be illuminated (i.e., illumination coefficient distribution information).

[0216] It is understood that the subject segmentation process can be either inputting the initial image into the subject segmentation model for subject segmentation, or inputting the image to be illuminated into the subject segmentation model for subject segmentation; this application does not limit this.

[0217] S703, the illumination template determination module obtains the light source parameter information and determines the target illumination distribution template corresponding to the light source parameter information from the preset illumination distribution template set.

[0218] For example, the lighting template set can be a collection of multiple pre-constructed lighting distribution templates, and the size of the lighting distribution templates can be the same as the size of the initial image to be processed; such as Figure 12 As shown, the lighting template set can include a basic lighting template subset and an extended lighting template subset. The basic lighting template subset includes multiple basic lighting distribution templates, and the extended lighting template subset includes multiple extended lighting distribution templates. The extended lighting distribution template is obtained by mapping the lighting color of the basic lighting distribution template. The light source direction corresponding to the lighting distribution template can be divided into front backlight direction and side backlight direction.

[0219] The basic illumination distribution template can be either real illumination distribution information obtained by illuminating a specified solid-color background with a real light source, or simulated illumination distribution information obtained by simulating a virtual light source illuminating a specified solid-color background using an optical model. Existing technologies can be used to obtain the basic illumination distribution template, which will not be elaborated upon here. Then, after obtaining a certain number of basic illumination distribution templates, illumination color conversion is performed based on a preset illumination color mapping relationship and the basic illumination distribution templates to obtain the extended illumination distribution templates corresponding to those templates.

[0220] Understandable, Figure 12 The examples provided are merely illustrative examples of several lighting distribution templates and do not represent the total number of templates included in the lighting template set. Furthermore, the basic lighting template subset may include a first lighting template subset and a second lighting template subset. The first lighting template subset includes multiple basic light source distribution templates corresponding to natural light sources, and the second lighting template subset includes multiple basic lighting distribution templates corresponding to artificial light sources.

[0221] S704, the illumination coefficient determination module determines the original outline information of the target subject based on the outline position information of the target subject.

[0222] The outline position information of the target subject can include the coordinate information of the outline position points in the target subject. Therefore, the original outline information of the target subject can be determined based on the coordinate information of the outline position points.

[0223] S705, the illumination coefficient determination module determines the boundary contour information based on the original contour information of the target subject, and determines the subject edge region in the initial image based on the original contour information and the boundary contour information.

[0224] For example, in the case where the light source direction is both forward and backlighting, such as... Figure 13aAs shown, the boundary contour information of the target subject can include the coordinate information of multiple contour position points on the inner edge contour. The inner edge contour can be the subject edge contour that moves a certain distance from the outermost contour to the center of the target subject. Correspondingly, the subject edge region includes the image region formed between the original contour and the inner edge contour.

[0225] For example, in the case where the light source direction is side-backlighting, such as... Figure 13b As shown, the boundary contour information of the target subject can include the coordinate information of multiple contour position points on the projected edge contour. The projected edge contour can be the subject edge contour obtained by projecting the original contour along the light source direction. Correspondingly, the subject edge region includes the image region formed between the original contour and the projected edge contour.

[0226] S706, the illumination coefficient determination module divides the subject edge region in the initial image to obtain multiple subject sub-image regions.

[0227] For example, the main sub-image region can be divided into four types of image partitions: Type I partition, Type II partition, Type III partition, and Type IV partition. Among them, such as... Figure 14 As shown, the first type of partition corresponds to the illumination coefficient change curve A (e.g., the illumination coefficient is a fixed value of 0). This type of image region can be called a backlit region, that is, a region where the light does not reach; for example, the overlapping region between the original image region (i.e., the image region within the original outline) and the projected region (i.e., the image region within the projection edge outline) of the subject. The second type of partition corresponds to the illumination coefficient change curve B (e.g., the illumination coefficient changes according to a cosine function, the illumination coefficient gradually decreases, and the region changes from 1 to 0 from the start position to the end position). This type of image region can be called a sidelit region, that is, the light intensity gradually decreases along the direction of the light source. The third type of partition corresponds to the illumination coefficient change curve C (e.g., the illumination coefficient changes according to a sine function, the illumination coefficient gradually increases, and the region changes from 0 to 1 from the start position to the end position). This type of image region can be called a gradient region, that is, the light intensity gradually increases along the direction of the light source. The fourth type of partition corresponds to the illumination coefficient change curve D (e.g., the illumination coefficient is a fixed value of 1). This type of image region can be called a normally lit region, that is, a region directly illuminated by the light.

[0228] For example, the illumination coefficient variation curve B can be represented as:

[0229]

[0230] For example, the illumination coefficient variation curve C can be represented as:

[0231]

[0232] Where, θ 1n θ represents the light intensity coefficient at location n in the second type of partition. 2n x represents the light intensity coefficient at location n in the third type of partition. s This represents the coordinate information of the starting position of the region, x e This indicates the coordinate information of the end position of the region, x n This represents the coordinate information at location n in the region, and the threshold value for adjusting the intensity of the lighting rendering.

[0233] It is understandable that the region position n, the region start position, and the region end position all correspond to a certain pixel point in the subject sub-image region. The region start position refers to the starting pixel point of the subject sub-image region along the direction of illumination coefficient change, and the region end position refers to the ending pixel point of the subject sub-image region along the direction of illumination coefficient change. The region position n refers to any pixel point in the subject sub-image region along the direction of illumination coefficient change. The direction of illumination coefficient change can be preset. For example, the direction of illumination coefficient change can be from the left side of the subject to the right side of the subject. Alternatively, the direction of illumination coefficient change can be the illumination direction of the simulated light source.

[0234] It should be noted that, Figure 14 The diagram merely provides one possible representation of the illumination coefficient variation curves B and C, and this illustrative representation does not constitute a limitation on the specific scope of protection of this application.

[0235] S707, the illumination coefficient determination module generates illumination coefficient distribution information based on the illumination coefficient change curve of the background region in the initial image and the illumination coefficient change curve of each subject sub-image region.

[0236] The background area is illuminated according to the original light intensity in the target illumination distribution template; therefore, the illumination coefficient variation curve for the background area is illumination coefficient variation curve D. The illumination coefficient variation curve for the subject sub-image area is determined based on the partition type to which the subject sub-image area belongs. The partition type is determined based on the light source direction and the relative positional relationship between the subject sub-image area and the background area. Therefore, if the subject sub-image area belongs to the first type of partition, the illumination coefficient variation curve is illumination coefficient variation curve A; if it belongs to the second type of partition, the illumination coefficient variation curve is illumination coefficient variation curve B; if it belongs to the third type of partition, the illumination coefficient variation curve is illumination coefficient variation curve C; and if it belongs to the fourth type of partition, the illumination coefficient variation curve is illumination coefficient variation curve D.

[0237] For example, in the case where the light source direction is both forward and backlighting, such as... Figure 15a As shown, taking the direction of illumination coefficient change from the left side of the subject to the right side of the subject as an example, the area formed by the left contour of the original contour and the left contour of the inner edge contour can be regarded as the second type of partition. The illumination coefficient change curve corresponding to the subject sub-image area is illumination coefficient change curve B, that is, the illumination coefficient gradually changes from 1 to 0. The contour position point on the original contour is used as the starting position of the area, and the contour position point on the inner edge contour is used as the ending position of the area. The area formed by the right contour of the original contour and the right contour of the inner edge contour can be regarded as the third type of partition. The illumination coefficient change curve corresponding to the subject sub-image area is illumination coefficient change curve C, that is, the illumination coefficient gradually changes from 0 to 1. The contour position point on the inner edge contour is used as the starting position of the area, and the contour position point on the original contour is used as the ending position of the area. In addition, the area inside the inner edge contour can be regarded as the first type of partition, that is, the illumination will not reach this area, and the illumination coefficient is a fixed value of 0.

[0238] For example, in the case where the light source direction is side-backlighting, such as... Figure 15b As shown, taking the direction of illumination coefficient change from the left side of the subject to the right side of the subject as an example, the area between the left contour of the original contour and the left contour of the projected edge contour can be regarded as the second type of partition. The illumination coefficient change curve corresponding to the subject sub-image area is illumination coefficient change curve B, that is, the illumination coefficient gradually changes from 1 to 0. The contour position point on the original contour is taken as the starting position of the area, and the contour position point on the projected edge contour is taken as the ending position of the area. The area between the right contour of the original contour and the right contour of the projected edge contour can be regarded as the third type of partition. The illumination coefficient change curve corresponding to the subject sub-image area is illumination coefficient change curve C, that is, the illumination coefficient gradually changes from 0 to 1. The contour position point on the projected edge contour is taken as the starting position of the area, and the contour position point on the original contour is taken as the ending position of the area. In addition, the area between the left contour of the projected edge contour and the right contour of the original contour can be regarded as the first type of partition, that is, the illumination will not reach this area, and the illumination coefficient is a fixed value of 0.

[0239] It should be noted that, in order to more clearly and intuitively illustrate the different types of regional divisions of the main body's edge area, the above... Figures 15a to 15b In this application, the edge outline of the subject being photographed is schematically represented as an "elliptical shape," but this schematic representation does not constitute a limitation on the specific scope of protection of this application.

[0240] In addition, such as Figure 15cAs shown, a magnified schematic diagram of the arm and leg area of ​​the target subject is provided, further illustrating the different types of area division in a certain block of the subject's edge region. Here, 1 represents the first type of partition, 2 represents the second type of partition, 3 represents the third type of partition, and 4 represents the first type of partition.

[0241] S708, the image fusion module performs element-wise multiplication of the target illumination distribution template and the illumination coefficient distribution information to obtain an illumination-weighted rendering image.

[0242] Specifically, the image fusion module acquires the target illumination distribution template sent by the illumination template determination module and the illumination coefficient distribution information sent by the illumination coefficient determination module. Then, it establishes a correspondence between each pixel in the target illumination distribution template and each pixel in the illumination coefficient distribution information. For example, in the same coordinate system, the coordinate information of the first pixel in the target illumination distribution template is the same as that of the second pixel in the illumination coefficient distribution information. Based on this correspondence, it calculates the product of the illumination intensity of the first pixel in the target illumination distribution template and the illumination coefficient of the second pixel in the illumination coefficient distribution information to obtain an illumination weighted rendering map. The illumination weighted rendering map is used to characterize the illumination distribution information to be added, obtained by weighting the illumination intensity of each pixel in the target illumination distribution template based on the illumination coefficient distribution information.

[0243] For example, the process of determining the light-weighted rendering map can be represented as follows:

[0244] P i，j =k i，j N i，j

[0245] Among them, P i，j N represents the weighted pixel value of pixel (i, j) in the light-weighted rendering image. i，j k represents the pixel value of pixel (i, j) in the target illumination distribution template. i，j This represents the illumination coefficient of pixel (i, j) in the illumination coefficient distribution information.

[0246] S709, the image fusion module performs image fusion processing based on the above-mentioned image to be lit and the light-weighted rendering image to obtain the target lighting image.

[0247] Specifically, firstly, based on the light-weighted rendering map, the weighted pixel value of each pixel in the image to be lit is determined. This weighted pixel value is used to characterize the light intensity of the pixel in the light-weighted rendering map. Then, for each pixel in the image to be lit, the original pixel value and the weighted pixel value of the pixel are weighted and summed to obtain the fused pixel value of the pixel. The original pixel value is used to characterize the pixel color of the image to be lit. Next, based on the fused pixel values ​​of each pixel in the image to be lit, the target lighting image is generated.

[0248] For example, the image fusion process can be understood as a weighted summation of pixel values. The fused pixel value of each pixel in the target lighting image is calculated based on a preset image fusion formula; whereby the image fusion formula can be expressed as:

[0249] L i，j =min(n1M) i，j +n2P i，j C max )

[0250] Among them, L i，j M represents the merged pixel value of pixel (i, j). i，j P represents the original pixel value of pixel (i, j) in the image to be illuminated. i，j C represents the weighted pixel value of pixel (i, j) in the light-weighted rendering image. max n1 represents the maximum pixel value of pixel (i, j) in the target lighting image, n2 represents the fusion weight coefficient corresponding to the initial image, and n2 represents the fusion weight coefficient of the lighting weighted rendering image.

[0251] Next, the image fusion module sends the target lighting image to the target application. After receiving the target lighting image, the target application displays it so that the user can confirm whether the lighting effect of the target lighting image meets expectations.

[0252] It should be noted that the specific implementation process of the above image lighting processing can be implemented by a functional module in the application framework layer, a functional module in the hardware abstraction layer, or a functional module in the digital signal processor. This application does not limit this.

[0253] Furthermore, the specific implementation process of the above-mentioned image lighting processing can be used to light up an initial sample image (corresponding to the initial image) to generate target sample images (corresponding to the target lighting images) with diverse lighting effects; then, based on the sample image set, the model to be trained is iteratively trained to obtain the trained image recognition model. This ensures the accuracy of the image recognition model in recognizing images with different lighting effects. Alternatively, it can address situations where the background lighting effect is unsatisfactory due to unfavorable ambient light conditions, thus meeting the user's need for relighting the image.

[0254] For example, in the case where the light source direction is both forward and backlighting, such as... Figure 16aAs shown, after acquiring the initial image to be processed, the initial image undergoes color gamut adjustment to obtain the image to be lit; and the initial image undergoes subject segmentation to determine the contour position information of the target subject. Based on the contour position information of the target subject, the original contour information and inner edge contour information of the target subject are determined; based on the original contour information and inner edge contour information, the subject edge region and background region in the initial image are determined, and based on the illumination coefficient change curve of the background region and the illumination coefficient change curve corresponding to the partition type of each subject sub-image region in the subject edge region, illumination coefficient distribution information is generated. Based on the target illumination distribution template and the illumination coefficient distribution information, an element-wise multiplication calculation is performed to obtain the illumination weighted rendering map; where the target illumination distribution template is the illumination distribution template in the preset illumination distribution template set that corresponds to the light source parameter information of the simulated light source. Finally, image fusion processing is performed on the image to be lit and the illumination weighted rendering map to obtain the target lit image.

[0255] For example, in the case where the light source direction is side-backlighting, such as... Figure 16b As shown, after acquiring the initial image to be processed, the initial image undergoes color gamut adjustment to obtain the image to be lit; and the initial image undergoes subject segmentation to determine the contour position information of the target subject. Based on the contour position information of the target subject, the original contour information and projected edge contour information of the target subject are determined; based on the original contour information and projected edge contour information, the subject edge region and background region in the initial image are determined, and based on the illumination coefficient change curve of the background region and the illumination coefficient change curve corresponding to the partition type of each subject sub-image region in the subject edge region, illumination coefficient distribution information is generated. Based on the target illumination distribution template and the illumination coefficient distribution information, an element-wise multiplication calculation is performed to obtain the illumination weighted rendering map; where the target illumination distribution template is the illumination distribution template corresponding to the light source parameter information of the simulated light source in the preset illumination distribution template set. Finally, image fusion processing is performed on the image to be lit and the illumination weighted rendering map to obtain the target lit image.

[0256] Furthermore, when the initial image contains multiple subjects, determining which subjects in the initial image are the target subjects can be done automatically based on preset rules, selecting subjects of a specified type, or it can be triggered by the user to select at least one subject from multiple subjects. For example... Figure 17 As shown, the display is as follows Figure 17 As shown in (a) of the mobile phone desktop, the user clicks the camera application icon 501; the phone responds to the user's action and displays as shown in the image. Figure 17As shown in (b) of the image preview interface, the preview area 502 of the image preview interface displays the image data of the subject currently captured by the camera of the electronic device. At this time, the user can click the first lighting control 503 in the image preview interface; the mobile phone responds to the click operation and displays as shown in the image preview interface. Figure 17 The photo preview interface shown in (c) displays subject identifiers (such as object 1 and object 2) of multiple subjects included in the preview image below the preview area 502. Object 1 can correspond to a human figure in the image, and object 2 can correspond to a tree. The user can select at least one subject as the target subject by clicking on the subject identifier (such as object 1). In response to this operation, the phone sets object 1 as the target subject (i.e., the target object). After acquiring the next frame of the preview image to be displayed, it performs background lighting processing on the preview image based on the light source parameter information to obtain the target lit image. Next, the image is displayed as shown... Figure 17 As shown in (d) of the image preview interface, the preview area 502 displays the target lighting image. If the background lighting effect of the target lighting image meets the user's expectations, the user can click the camera control 504. The phone responds to the click operation, captures the image through the camera, performs background lighting processing on the captured image, obtains the target lighting image corresponding to the captured image, and stores the target lighting image. The user can quickly view the target lighting image corresponding to the captured image from the gallery by clicking the photo viewing control in the lower left corner of the image preview interface.

[0257] It should be noted that when there are multiple target subjects, the illumination coefficient variation curves corresponding to the subject sub-image regions in each target subject can be determined by referring to the specific implementation process described above. Then, based on the illumination coefficient variation curves of the background region in the initial image and the illumination coefficient variation curves corresponding to the subject sub-image regions in each target subject, illumination coefficient distribution information is generated. Next, the illumination distribution template and the illumination coefficient distribution information are multiplied element-wise to obtain an illumination weighted rendering image. Finally, image fusion processing is performed based on the image to be illuminated and the illumination weighted rendering image to obtain the target illuminated image.

[0258] Scene 2

[0259] In this scenario, regarding the initial image acquisition process, the initial image can be a preview image requested by the camera application from the camera. For example, after the phone detects the user's click on the lighting control, it first applies lighting processing to the preview image to be displayed, and then displays the lit preview image in the preview area of ​​the photo preview interface, corresponding to the above. Figure 5 and Figure 6The diagram illustrates an application scenario for triggering lighting processing on a preview image. This lighting processing can be executed by a digital signal processor (DSP). Specifically, for subject lighting, the aforementioned key point extraction module, 3D model construction module, illumination distribution determination module, and image fusion module are integrated into the DSP. For background lighting, the aforementioned image preprocessing module, illumination template determination module, illumination coefficient determination module, and image fusion module are integrated into the DSP. Figure 18a As shown, after the phone detects a user's application launch trigger operation on the camera application (i.e., a click on the application icon, i.e., user operation 1), in response to the user operation, the camera application sends an image preview request to the camera service in the application framework layer; after receiving the image preview request, the camera service sends an image acquisition request to the camera HAL in the hardware abstraction layer; after receiving the image acquisition request, the camera HAL sends a camera driver instruction to the camera driver in the kernel layer; after receiving the camera driver instruction, the camera driver sends a camera control instruction to the camera to control the camera to acquire image data in real time and obtain the preview image to be displayed.

[0260] In other words, the camera application in the application layer calls the camera HAL in the hardware abstraction layer through the camera service in the application framework layer, thereby controlling the camera driver in the kernel layer to take pictures. Specifically, the camera application sequentially instructs the camera driver in the kernel layer through the camera service in the application framework layer and the camera HAL in the hardware abstraction layer, driving the camera of the electronic device to perform image acquisition and processing on the currently captured area to obtain a preview image.

[0261] After the camera captures a preview image in real time, it sends the preview image to the camera driver; the camera driver sends the preview image to the camera HAL; the camera HAL sends the preview image to the camera service; the camera service sends the preview image to the camera application; and the preview image is displayed in the preview area of ​​the camera application's photo preview interface. While the preview image is displayed in the preview area, it is also refreshed in real time. The refresh process for the preview image can be found in existing technologies and will not be elaborated upon here.

[0262] Next, after the phone detects the user's image lighting trigger operation (i.e., the click operation on the lighting control, i.e., user operation 2), it responds to the user operation and determines the light source parameter information; the camera application sends the light source parameter information to the camera service in the application framework layer; the camera service sends the light source parameter information to the camera HAL in the hardware abstraction layer; the camera HAL sends the light source parameter information to the illumination distribution determination module in the digital signal processor; and the camera HAL sends the preview image to be displayed to the digital signal processor.

[0263] Specifically, if the image lighting type is subject lighting, the key point extraction module performs key point extraction processing based on the preview image to obtain the spatial distribution information of key points on the target face; and sends this key point spatial distribution information to the stereo model construction module. The stereo model construction module performs 3D reconstruction processing based on the key point spatial distribution information to obtain a stereo model of the target face; and sends this stereo model of the face to the illumination distribution determination module. The illumination distribution determination module performs simulated lighting processing based on the light source parameter information and the stereo model of the face to obtain first illumination spatial distribution information and second illumination spatial distribution information; and sends the first illumination spatial distribution information and second illumination spatial distribution information to the image fusion module. The image fusion module performs image fusion processing based on the preview image, the first illumination spatial distribution information, and the second illumination spatial distribution information to obtain the first lighting image.

[0264] Specifically, if the image lighting type is background lighting, the image preprocessing module adjusts the color gamut of the initial image to obtain the image to be lit; and performs subject segmentation on the initial image to determine the contour position information of the target subject. The illumination template determination module determines the target illumination distribution template from a preset illumination distribution template set based on the light source parameter information of the simulated light source. The illumination coefficient determination module determines the subject edge region in the initial image based on the contour position information of the target subject, and generates illumination coefficient distribution information based on the illumination coefficient change curves corresponding to each subject sub-image region in the subject edge region. The image fusion module generates an illumination weighted rendering map based on the target illumination distribution template and the illumination coefficient distribution information; and performs image fusion processing on the image to be lit and the illumination weighted rendering map to obtain the first lighting image.

[0265] The image fusion module sends the first lighting image to the camera HAL in the hardware abstraction layer; the camera HAL sends the first lighting image to the camera service; the camera service sends the first lighting image to the camera application; the preview area of ​​the camera application's photo preview interface displays the first lighting image; the user can confirm whether the lighting effect of the first lighting image (such as the lighting effect of the subject or the lighting effect of the background) meets expectations, and decide whether to trigger the image shooting process through the photo control.

[0266] Furthermore, if the user confirms that the lighting effect of the first illuminated image meets expectations, they can trigger the image capture process via the camera control. For example... Figure 18bAs shown, after the mobile phone detects the user's photo-taking trigger operation (i.e., the click operation on the camera control, i.e., user operation 3), in response to the user operation, the camera application sends an image capture request to the camera service in the application framework layer; after receiving the image capture request, the camera service sends an image acquisition request to the camera HAL in the hardware abstraction layer; after receiving the image acquisition request, the camera HAL sends a camera driver instruction to the camera driver in the kernel layer; after receiving the camera driver instruction, the camera driver sends a camera control instruction to the camera to control the camera to acquire image data in real time and obtain the captured image.

[0267] After capturing an image in real time, the camera sends the captured image to the camera driver; the camera driver then sends the captured image to the camera HAL (Hydraulic Alignment). Since the captured image needs to be illuminated, the camera HAL can directly send the captured image to the digital signal processor (DSP). Furthermore, the camera HAL sends light source parameter information from the camera application to the DSP. The DSP performs illumination processing on the captured image based on the light source parameter information to obtain a second illuminated image. The process of determining the second illuminated image can refer to the specific implementation process of generating the first illuminated image described above, and will not be repeated here.

[0268] The digital signal processor sends a second lighting image to the camera HAL, which in turn sends the second lighting image to the camera service. The camera service then stores the second lighting image in a designated location, allowing it to be viewed through a gallery application.

[0269] In this way, when the shooting scene cannot meet the user's shooting needs, such as in a dimly lit outdoor scene or a dimly lit indoor environment, or when it is difficult to deploy a light source for background lighting, if the user needs to capture an image with a certain lighting effect (such as subject lighting or background lighting), a preview image (i.e., the original image captured by the camera in real time) can be displayed in the preview area of ​​the camera app's photo preview interface. At this time, the user can directly trigger the lighting processing of the preview image in the photo preview interface of the camera app according to the actual needs. Then, the target lighting image (i.e., the image after lighting processing of the preview image) corresponding to the preview image is displayed in the photo preview interface of the camera app. This allows the user to promptly confirm whether the lighting effect of the target lighting image displayed in the preview area meets expectations during the photo preview process. If the lighting effect meets expectations, the user can click the photo control to trigger the direct storage of the target lighting image corresponding to the captured image, without the need to separately store the un-lit image. Furthermore, after the photo is taken, there is no need to open the gallery app or a third-party photo editing app again to process the image lighting. In addition, the image lighting process is set in a digital signal processor, thereby improving the efficiency of generating target lighting images by leveraging the digital signal processor's image signal processing capabilities.

[0270] Understandably, the preview image can be first sent to the camera application via the camera HAL and camera service, and then the target application sends it to the digital signal processor via the camera service and camera HAL. However, since the preview image before lighting processing is directly displayed instead of the pre-processed preview image, after receiving the preview image to be displayed from the camera driver, the camera HAL can directly send the preview image to be displayed to the digital signal processor for image lighting processing to obtain the first lit image. This eliminates the process of first sending the preview image to be displayed sequentially to the camera service and camera application, and then having the camera application send the preview image to be displayed sequentially to the camera service and camera HAL. Furthermore, if the light source parameters do not change, the camera application does not need to repeatedly send the light source parameter information to the camera service and camera HAL sequentially.

[0271] Furthermore, if the user confirms that the lighting effect of the first lighting image does not meet expectations, then adjustments to the lighting effect of the preview image can be triggered through a specified control until an adjusted lighting image that meets the user's expectations is obtained, corresponding to the above. Figure 7 and Figure 8The illustration shows an application scenario where relighting of a preview image is triggered. For example... Figure 18c As shown, after detecting a user's click on the lighting control, and without detecting a user's click on the camera control, the camera continuously returns a preview image to be displayed to the camera HAL via the camera driver. Upon receiving the preview image, the camera HAL sends it to the digital signal processor (DSP). The DSP performs image lighting processing on the preview image to obtain a first lit image. The camera HAL then returns the first lit image to the camera application via the camera service, and the camera application displays the first lit image in the preview area.

[0272] Next, after detecting the user's relighting trigger operation (i.e., a click on the lighting control or human-computer interaction control, i.e., user operation 4), the phone responds to the user's operation and determines the adjusted light source parameter information. The target application sends the adjusted light source parameter information to the camera HAL via the camera service, and the camera HAL sends the adjusted light source parameter information to the digital signal processor. Furthermore, since the lighting effect of the preview image to be displayed needs to be adjusted based on the adjusted light source parameter information, the camera HAL sends the preview image to the digital signal processor. The digital signal processor performs lighting processing on the preview image to be displayed based on the adjusted light source parameter information to obtain the adjusted lighting image; and sends the adjusted lighting image to the camera HAL, which then sends the adjusted lighting image to the camera service; the camera service sends the adjusted lighting image to the camera application; the preview area of ​​the camera application's photo preview interface displays the adjusted lighting image; the user can confirm whether the lighting effect of the adjusted lighting image meets expectations and decide whether to trigger the image shooting process through the photo control.

[0273] The adjusted light source parameter information can be referred to the specific implementation process described above, and the process of determining the adjusted lighting image can be referred to the specific implementation process of generating the first lighting image described above, which will not be repeated here.

[0274] Scene 3

[0275] In this scenario, regarding the acquisition of the initial image, the initial image can also be a historically captured image selected by the user from a gallery application, corresponding to the above. Figure 9 This diagram illustrates an application scenario where lighting is applied to a gallery image. For example, if the user selects an unlit image from the gallery, the process of applying lighting to that image can still be executed by a digital signal processor. Figure 18dAs shown, after the phone detects the user's photo-taking trigger operation (i.e., the click operation on the camera control, i.e., user operation 5), in response to the user operation, the camera application sends an image capture request to the camera service; after receiving the image capture request, the camera service sends an image acquisition request to the camera HAL; after receiving the image acquisition request, the camera HAL sends a camera driver instruction to the camera driver; after receiving the camera driver instruction, the camera driver sends a camera control instruction to the camera to control the camera to acquire image data in real time and obtain the captured image.

[0276] After the camera captures an image in real time, it sends the captured image to the camera driver; the camera driver sends the captured image to the camera HAL; the camera HAL sends the captured image to the camera service; and the captured image data is stored in a specified location, so that the captured image can be viewed through the gallery application and selected as a gallery image to be lit.

[0277] Then, when the user triggers the opening of the gallery application, the user can select a gallery image to be processed from multiple gallery images. If the phone detects the user's image lighting trigger operation (i.e., a click on the lighting control, i.e., user operation 6), it responds to the user operation by acquiring the light source parameter information. The gallery application sends the gallery image to be processed and the light source parameter information to the image processing service in the application framework layer. The image processing service sends the gallery image to be processed and the light source parameter information to the camera HAL in the hardware abstraction layer. Then, the camera HAL sends the gallery image to be processed to the key point extraction module and image fusion module in the digital signal processor, and sends the light source parameter information to the illumination distribution determination module. The key point extraction module performs key point extraction processing based on the gallery image to obtain the spatial distribution information of the key points of the target face; and sends this key point spatial distribution information to the stereo model construction module. The stereo model construction module performs 3D reconstruction processing based on the key point spatial distribution information to obtain a stereo model of the target face; and sends this stereo model of the face to the illumination distribution determination module. The illumination distribution determination module performs simulated lighting processing based on light source parameter information and a 3D face model to obtain first and second illumination spatial distribution information; it then sends the first and second illumination spatial distribution information to the image fusion module. The image fusion module performs image fusion processing based on the image in the image library, the first and second illumination spatial distribution information, to obtain a third illuminated image.

[0278] The image fusion module sends a third lighting image to the camera HAL in the hardware abstraction layer; the camera HAL sends the third lighting image to the image processing service; the image processing service sends the third lighting image to the gallery application; and the gallery application displays the third lighting image on the image editing interface. If the currently displayed third lighting image meets the user's expectations, the user can trigger saving the third lighting image. If the currently displayed target lighting image does not meet the user's expectations, the user can readjust the image lighting parameters to trigger regeneration of the target lighting image; alternatively, the user can also trigger the acquisition of new gallery images and the generation of new target lighting images corresponding to those images. The specific implementation process can be referred to the above process and will not be elaborated further here.

[0279] Next, after detecting the user's confirmation trigger (i.e., clicking the save control, or user action 7), the phone responds to this user action by sending a lighting image storage request to the image processing service. The image processing service then stores the third lighting image in the specified location, allowing the user to view this third lighting image through the gallery app.

[0280] In this way, by adding an image lighting control to the gallery app, after an image is captured, if the user views the captured images in the gallery app and finds that the images were not properly lit, or that the lighting effect of the lit images does not meet expectations, the user can directly trigger the lighting process within the gallery app at any time to obtain the desired lit image. This ensures that the user's needs for image lighting processing are met in different application scenarios. Furthermore, triggering image lighting processing directly within the gallery app eliminates the need to open third-party image editing applications. Additionally, by setting the image lighting processing within the digital signal processor (DSP), the processing capabilities of the DSP for image signals are leveraged to improve the efficiency of generating the target lit image.

[0281] Understandably, users can choose either of the two implementation methods mentioned above based on their actual scenario, providing them with more diverse ways to trigger image lighting processing.

[0282] Additionally, a preview image refers to the image displayed in real-time on the camera app's preview screen before the user clicks the shutter button; a captured image refers to the image data captured by the camera when the user clicks the shutter button. Generally, after opening the camera app but before the user clicks the shutter button, the image preview stage begins, displaying the preview image on the camera app's preview screen, and the preview image is continuously refreshed. Upon detecting the user clicking the shutter button, the image capture stage begins, where the camera captures the image. Next, after opening the gallery app, the image viewing stage begins, displaying historically captured images on the gallery app's display screen.

[0283] In the embodiments provided in this application, before the user clicks the camera control, i.e., during the image preview stage, clicking the lighting control triggers lighting processing on the preview image, and the target lighting image corresponding to the preview image is displayed on the camera application's photo preview interface. At this time, the user can determine whether the lighting effect of the displayed target lighting image meets expectations. If the user confirms that the image lighting effect meets expectations, clicking the camera control triggers the storage of the target lighting image corresponding to the captured image. Specifically, when a user click operation on the first lighting control (i.e., the lighting control on the camera application's display interface) is detected, the preview image to be displayed is first lit to obtain a first lighting image; then, the first lighting image corresponding to the preview image is displayed on the photo preview interface of the camera application. Then, when a user clicks the camera control, the camera captures a captured image; and the captured image is lit to obtain a second lighting image, which is stored. This second lighting image can then be viewed in the gallery application.

[0284] Additionally, after opening the Gallery app and entering the image viewing stage, users can click the lighting control to trigger lighting processing on the gallery image (such as historically captured image data). The target lighting image corresponding to the gallery image will then be displayed on the Gallery app's interface. Users can then assess whether the displayed target lighting effect meets their expectations. If the user confirms that the image lighting effect meets expectations, they can click the save control to save the target lighting image corresponding to the gallery image. Specifically, upon detecting a user click on the second lighting control (i.e., the lighting control on the Gallery app's interface), the selected gallery image is first lit to obtain a third lighting image; this third lighting image is then displayed on the Gallery app's interface. Finally, upon detecting a user click on the save control, the third lighting image corresponding to that gallery image is saved, allowing users to view this third lighting image within the Gallery app.

[0285] It should be noted that the image lighting process can also be executed by the application processor, specifically by functional modules in the application framework layer or by functional modules in the hardware abstraction layer. Specifically, when image lighting is performed by functional modules in the application framework layer, if the image lighting type is background lighting, the application framework layer can include an image preprocessing module, a lighting template determination module, a lighting coefficient determination module, and an image fusion module. If the image lighting type is subject lighting, the application framework layer can include a key point extraction module, a 3D model construction module, a lighting distribution determination module, and an image fusion module. In response to the image lighting trigger operation, the camera application sends the initial image to be processed and light source parameter information to the image processing service. The image processing service performs image lighting processing based on the initial image and light source parameter information to obtain the target lit image. Then, the image processing service returns the target lit image to the camera application, and the target lit image is displayed in the preview area of ​​the camera application's photo preview interface. When image lighting processing is performed by functional modules in the Hardware Abstraction Layer (HAL), if the image lighting type is background lighting, the camera HAL may include an image preprocessing module, a lighting template determination module, a lighting coefficient determination module, and an image fusion module. If the image lighting type is subject lighting, the camera HAL may include a key point extraction module, a 3D model construction module, a lighting distribution determination module, and an image fusion module. In response to the image lighting trigger operation, the camera application sends the initial image to be processed and light source parameter information to the camera service; the camera service sends the initial image to be processed and light source parameter information to the camera HAL; the camera HAL performs image lighting processing based on the initial image and light source parameter information to obtain the target lighting image; then, the camera HAL returns the target lighting image to the camera service, the camera service returns the target lighting image to the camera application, and the preview area of ​​the camera application's photo preview interface displays the target lighting image. The generation process of the target lighting image can be referred to as follows: Figure 11 The specific implementation process is shown below, or refer to the following example. Figure 20 The specific implementation process shown will not be elaborated here.

[0286] In some example embodiments, the electronic device can provide the user with two image lighting functions, such as a subject lighting function and a background lighting function. If the user triggers the selection of the background lighting function, the electronic device executes the relevant processing flow for lighting the background area in the initial image, as detailed in steps S701 to S709 above; if the user triggers the selection of the subject lighting function, the electronic device executes the relevant processing flow for lighting the target object in the initial image, as detailed in steps S801 to S808 below; wherein, the user needs to trigger the selection of which image lighting function, such as... Figure 19aAs shown, in one scenario, the image lighting type is pre-set by opening the settings app and entering the settings interface. For example, the image lighting type is set to subject lighting, meaning the subject lighting switch is turned on. Then, during image lighting processing, the image lighting type is determined based on the image lighting type setting information. In another scenario, such as... Figure 19b As shown, the user clicks the camera application icon 501; the phone responds to the user's operation and displays a photo preview interface. The preview area 502 in the photo preview interface displays the image data of the subject currently captured by the electronic device's camera. At this time, the user can click the first lighting control 503 in the photo preview interface to trigger image lighting processing; wherein, the first lighting control 503 may include a first sub-control 5031 (a control for triggering subject lighting) and a second sub-control 5032 (a control for triggering background lighting). If a user click operation on the second sub-control 5032 is detected, it is determined that the user has selected to use the background lighting function. Figure 19c As shown, if a user click on the first sub-control 5031 is detected, it is determined that the user has selected to use the subject lighting function. This allows the user to flexibly choose which image lighting function to trigger on the photo preview interface, based on the preview image displayed there.

[0287] It is understandable that image lighting types can include subject lighting and background lighting. For subject lighting, the image area to which the lighting effect is added must at least include the image area containing the target subject (e.g., a face). The target lighting image is the subject lighting image, the first lighting image is the subject lighting image corresponding to the preview image, the second lighting image is the subject lighting image corresponding to the captured image, and the third lighting image is the subject lighting image corresponding to the image selected in the gallery application. For background lighting, the image area to which the lighting effect is added must at least include the image background area; to improve the lighting effect, the subject edge area may also be included. The target lighting image is the background lighting image, the first lighting image is the background lighting image corresponding to the preview image, the second lighting image is the background lighting image corresponding to the captured image, and the third lighting image is the background lighting image corresponding to the image selected in the gallery application.

[0288] Scene 4

[0289] For images where the lighting type is subject lighting, such as Figure 20 As shown, the specific implementation process of generating the target lighting image corresponding to the initial image may include the following steps:

[0290] S801, the key point extraction module uses a pre-trained face key point detection model to perform key point extraction processing on the initial image to obtain the spatial distribution information of key points of the target face.

[0291] The target application can include any one of a camera application, a gallery application, or other applications with shooting capabilities. The target application sends an initial image to be lit to the keypoint extraction module. The keypoint extraction module acquires the initial image, which includes the image region of the target face. For example, in one scenario, the initial image can be a preview image to be displayed on the camera application's preview interface; the phone detects the user's trigger operation on the camera application's icon, launches the camera application, and captures the preview image through the camera. In another scenario, the initial image can be a captured image; the phone detects the user's trigger operation on the camera application's shooting control and captures the captured image through the camera. Yet another scenario, the initial image can be a historical image from the gallery application; the phone detects the user's trigger operation on the gallery application and determines the photo selected by the user from the gallery application interface as the initial image. After acquiring the initial image, the target application sends it to the keypoint extraction module and the image fusion module.

[0292] For example, the facial landmark detection model is obtained by iteratively updating the parameters of a neural network model based on a set of facial image samples. The facial landmark detection model not only learns to recognize the planar coordinates of facial landmarks, but also learns to recognize the depth information of facial landmarks. That is, it learns the spatial distribution of key feature points in different parts of the target face. Therefore, the positional information of each facial landmark in the spatial distribution information includes not only the planar coordinate information of the facial landmark, but also the depth information of the facial landmark.

[0293] In this process, after the key point extraction module generates the spatial distribution information of key points of the target face, it sends the spatial distribution information of key points to the stereo model construction module so that the stereo model construction module can construct a stereo model of the target face based on the spatial distribution information of key points.

[0294] S802, the stereo model construction module uses the neighborhood growth method to perform three-dimensional reconstruction based on the spatial distribution information of key points of the target face to obtain the stereo model of the target face; taking the construction of a face unit surface by three adjacent face key points as an example, the stereo model of the face includes multiple triangular facets.

[0295] For example, from multiple facial keypoints in the keypoint spatial distribution information, a starting growth point and an ending growth point are determined; for instance, the facial keypoint with the greatest depth is determined as the starting growth point, and the facial keypoints on the facial boundary contour line are determined as the ending growth points. The starting growth point is used as the first selected current growth point, and adjacent growth points are determined, establishing connections between the current growth point and its adjacent growth points; adjacent growth points are used as the next selected current growth point, and adjacent growth points are determined, establishing connections between the next current growth point and its adjacent growth points; this process continues until the current growth point is the ending growth point and each facial keypoint has been selected as a current growth point; the keypoint spatial distribution information with established connections is used to determine the 3D facial model of the target face.

[0296] For example, such as Figure 21 The diagram illustrates the detailed implementation process of obtaining a 3D face model by starting growth from the initial growth point and ending growth at the termination growth point. (See also...) Figure 21 (a) shows the spatial distribution information of key points on the target face; see also Figure 21 In (b) of the keypoint spatial distribution information, a starting growth point A0 (i.e., the keypoint at the highest point of the bridge of the nose) and multiple ending growth points B (i.e., the facial boundary keypoints, specifically the light gray keypoints on the outermost edge circled by the dashed line) are determined from multiple facial keypoints. See also Figure 21 In step (c), the initial growth point A0 is selected as the first current growth point. The adjacent growth points (e.g., A1 to A6) are then determined, and the connection between the current growth point and its adjacent growth points is established. See also... Figure 21 In step (d), the adjacent growth point is selected as the next current growth point, the adjacent growth points of the next current growth point are determined, and the connection relationship between the next current growth point and the adjacent growth points is established. See also Figure 21 In (e), until the current growth point is the termination growth point B and each facial key point is selected as a current growth point; the spatial distribution information of the key points after the connection relationship is established is determined as the facial stereo model of the target face.

[0297] In this module, after the 3D model construction module constructs a 3D model of the target face, it sends the 3D model to the illumination distribution determination module. The illumination distribution determination module then performs simulated lighting processing on the 3D model based on the light source parameter information of the simulated light source to obtain the spatial distribution information of illumination with different reflection types.

[0298] S803, the illumination distribution determination module determines the target illumination parameters of each triangular facet in the 3D face model based on the light source parameter information of the simulated lighting source.

[0299] After determining the light source parameter information of the simulated lighting source, the target application sends the light source parameter information to the illumination distribution determination module. The light source parameter information includes at least one of the following: light source type, light source luminous intensity, light source color, light source direction, light source center position information, illumination attenuation coefficient, and specular reflectance coefficient.

[0300] In some example embodiments, the light source parameter information for the simulated lighting source can be default light source parameter information. In other example embodiments, the light source parameter information can also be the light source parameter information determined by the mobile phone upon detecting the user's trigger operation on the lighting control, entering the lighting parameter setting interface, and based on the user's parameter setting confirmation information; specifically, the light source parameter information can be the light source parameter information determined by the user's confirmation operation based on automatically recommended light source parameter items, or it can be the user-defined light source parameter information. In still other example embodiments, the light source parameter information can also be the light source parameter information obtained by the mobile phone upon detecting the user's trigger operation on a specified control, entering the human-computer interaction interface, and matching parameter values ​​based on the user's input description of the desired lighting effect.

[0301] The target application sends the light source parameter information to the illumination distribution determination module, so that the illumination distribution determination module can determine the illumination spatial distribution information of the target face based on the light source parameter information.

[0302] For example, each triangular facet is treated as a face unit, and for each face unit, the target illumination parameter information corresponding to that face unit is determined. The target illumination parameter information includes the light source incident normal vector and the face unit normal vector. The light source incident normal vector of different triangular facets can be different, and the face unit normal vector of different triangular facets can also be different.

[0303] For example, such as Figure 22 The diagram illustrates how to determine the target illumination parameters of any triangular facet using a triangular facet as the smallest computational unit in a 3D face model. (See also...) Figure 22 In (a) of the 3D face model, the direction of the unit plane normal vector is different for different triangular facets; the unit plane normal vector can be determined based on the 3D coordinate information of the triangular facet, and its direction is perpendicular to the plane containing the triangular facet. See also Figure 22 In the diagram (b), a cross-sectional view of the three-dimensional face model is shown. L indicates the direction of the incident normal vector of the light source for a certain triangular facet, N indicates the direction of the unit facet normal vector for a certain triangular facet, V indicates the observation direction, and R indicates the direction of the specular reflection of a certain triangular facet.

[0304] S804, the illumination distribution determination module determines the specular reflection illumination intensity of each triangular facet in the 3D facet model based on the first light intensity calculation formula and the target illumination parameter information corresponding to that triangular facet.

[0305] For example, the formula for calculating the first light intensity can be expressed as:

[0306]

[0307] Among them, c s The light intensity of the specular reflection from a triangular facet is represented by k, and the light attenuation coefficient is represented by c. l Indicates the luminous intensity of a light source, m s High reflectance Represents the normal vector of the observation direction. Let represent the unit face normal vector of a given triangular facet. Let m represent the incident normal vector of a light source on a certain triangular facet. g This represents the skin reflectivity of the target face.

[0308] It is understandable that the normal vector of the observation direction, the skin reflectivity of the target face, the light attenuation coefficient, the light intensity of the light source, and the specular reflectivity are all known parameters.

[0309] S805, the illumination distribution determination module generates the first illumination spatial distribution information corresponding to the high-reflection light intensity of each triangular facet in the face stereo model.

[0310] Specifically, firstly, for each triangular facet in the 3D face model, based on the first mapping relationship and the specular reflection intensity corresponding to that triangular facet, the first pixel value of each pixel in that triangular facet is determined; each first pixel value is used to characterize the specular reflection color of any pixel in a certain triangular facet. Then, based on the first pixel values ​​of each pixel in the 3D face model, the first illumination spatial distribution information corresponding to the specular reflection is generated.

[0311] For example, the first mapping relationship includes the correspondence between the intensity of specular reflected light and the color of specular reflected light. Since the difference in specular reflected light intensity among pixels in a given triangular facet is relatively small, the specular reflected light color of each pixel in the given triangular facet can be considered the same; therefore, the first pixel value of each pixel in the given triangular facet can be considered the same.

[0312] For example, the first illumination spatial distribution information is used to characterize the distribution of specular reflection light color of each pixel in the 3D face model. In some example embodiments, considering that the 3D face model is composed of multiple triangular facets, due to the discreteness of the triangular facets, the illumination effect shown by the initially formed initial illumination spatial distribution information is not smooth and uniformly distributed, but rather appears as patchy spots. Therefore, in order to reduce the transitional differences in specular reflection light intensity (corresponding to the first pixel value) between adjacent triangular facets, thereby ensuring that the illumination effect shown by the first illumination spatial distribution information is smoother, the initial illumination spatial distribution information corresponding to the specular reflection is first generated based on the first pixel value of each pixel in the 3D face model; then, pixel smoothing processing is performed on the initial illumination spatial distribution information to obtain the first illumination spatial distribution information.

[0313] S806, the illumination distribution determination module determines the diffuse reflection illumination intensity of each triangular facet in the 3D facet model based on the second light intensity calculation formula and the target illumination parameter information corresponding to that triangular facet.

[0314] For example, the formula for calculating the second light intensity can be expressed as:

[0315]

[0316] Among them, c d Let k represent the diffuse reflection intensity of a triangular facet, and c represent the light attenuation coefficient. l Indicates the luminous intensity of a light source, m d Diffuse reflectance coefficient Let represent the unit face normal vector of a given triangular facet. This represents the incident normal vector of the light source for a certain triangular facet.

[0317] It is understandable that the light attenuation coefficient, the luminous intensity of the light source, and the diffuse reflection coefficient are all known parameters. Furthermore, considering that the diffuse reflection direction produced by each triangular facet is relatively dispersed, the diffuse reflection intensity produced by adjacent triangular facets may be projected onto a certain triangular facet. Therefore, the diffuse reflection intensity of a certain triangular facet can be determined by a combination of the diffuse reflection intensity produced by the triangular facet itself and the diffuse reflection intensity produced by adjacent triangular facets.

[0318] S807, the illumination distribution determination module generates second illumination spatial distribution information corresponding to diffuse reflection based on the diffuse reflection intensity of each triangular facet in the face stereo model.

[0319] Specifically, firstly, for each triangular facet in the 3D face model, based on the second mapping relationship and the diffuse illumination intensity corresponding to that triangular facet, the second pixel value of each pixel in that triangular facet is determined; each second pixel value is used to characterize the diffuse illumination color of any pixel in a certain triangular facet. Then, based on the second pixel values ​​of each pixel in the 3D face model, the second illumination spatial distribution information corresponding to diffuse reflection is generated.

[0320] For example, the second mapping relationship includes the correspondence between diffuse light intensity and diffuse light color. Since the difference in diffuse light intensity among pixels in a given triangular facet is relatively small, the diffuse light color of each pixel in the given triangular facet can be considered the same; consequently, the second pixel value of each pixel in the given triangular facet can be considered the same.

[0321] For example, the second illumination spatial distribution information is used to characterize the distribution of diffuse light color of each pixel in the 3D face model. In some example embodiments, considering that the 3D face model is composed of multiple triangular facets, due to the discreteness of the triangular facets, the illumination effect shown by the initially formed initial illumination spatial distribution information is not smooth and uniformly distributed, but rather appears as patchy spots. Therefore, in order to reduce the transitional differences in diffuse light intensity (corresponding to the second pixel value) between adjacent triangular facets, thereby ensuring that the illumination effect shown by the second illumination spatial distribution information is smoother, the initial illumination spatial distribution information corresponding to diffuse reflection is first generated based on the second pixel value of each pixel in the 3D face model; then, pixel smoothing processing is performed on the initial illumination spatial distribution information to obtain the second illumination spatial distribution information.

[0322] In this process, after the illumination distribution determination module generates the first illumination spatial distribution information and the second illumination spatial distribution information, it sends the first illumination spatial distribution information and the second illumination spatial distribution information to the image fusion module, so that the image fusion module can perform image fusion processing based on the first illumination spatial distribution information and the second illumination spatial distribution information to obtain the target illumination image.

[0323] S808, the image fusion module performs image fusion processing based on the initial image, the first illumination spatial distribution information, and the second illumination spatial distribution information to obtain the target illumination image.

[0324] Specifically, firstly, based on the first illumination spatial distribution information, the first pixel value of the target pixel in the initial image is determined; and secondly, based on the second illumination spatial distribution information, the second pixel value of the target pixel in the initial image is determined; the target pixel is any pixel on the 3D face model; then, the original pixel value, the first pixel value, and the second pixel value of the target pixel are weighted and summed to obtain the fused pixel value of the target pixel; wherein, the original pixel value is used to characterize the pixel color of the initial image; next, based on the original pixel values ​​of each non-target pixel in the initial image and the fused pixel values ​​of each target pixel, a target lighting image is generated; non-target pixels are pixels in the initial image that have not been lit.

[0325] For example, the image fusion process can be understood as a weighted summation of pixel values. The fused pixel value of each pixel in the target lighting image is calculated based on a preset image fusion formula; whereby the image fusion formula can be expressed as:

[0326] K i，j =min(m1C i，j +m2B i，j +m3M i，j C max )

[0327] Among them, K i，j C represents the merged pixel value of pixel (i, j). i，j B represents the original pixel value of pixel (i, j) in the initial image. i，j M represents the first pixel value of pixel (i, j). i，j C represents the second pixel value of pixel (i, j). max m1 represents the maximum pixel value of pixel (i, j) in the target lighting image, m2 represents the fusion weight coefficient corresponding to the initial image, m3 represents the fusion weight coefficient of the specular reflection lighting color, and m3 represents the fusion weight coefficient of the diffuse reflection lighting color.

[0328] It is understandable that for non-target pixels in the initial image, the first pixel value is equal to zero, the second pixel value is equal to zero, therefore, the fused pixel value of the non-target pixel is equal to the original pixel value.

[0329] Next, the image fusion module sends the target lighting image to the target application. After receiving the target lighting image, the target application displays it so that the user can confirm whether the lighting effect of the target lighting image meets expectations.

[0330] It should be noted that the specific implementation process of the above image lighting processing can be implemented by a functional module in the application framework layer, a functional module in the hardware abstraction layer, or a functional module in the digital signal processor. This application does not limit this.

[0331] Furthermore, the specific implementation process of the above-mentioned image lighting processing can be used to light up an initial sample image (corresponding to the initial image) to generate target sample images (corresponding to the target lighting images) with diverse lighting effects; then, based on the sample image set, the model to be trained is iteratively trained to obtain the trained image recognition model. This ensures the accuracy of the image recognition model in recognizing images with different lighting effects. Alternatively, it can address situations where the background lighting effect is unsatisfactory due to unfavorable ambient light conditions, thus meeting the user's need for relighting the image.

[0332] In the image processing method provided in the embodiments of this application, the process of lighting the subject is as follows: Figure 23 As shown, firstly, facial key points are extracted from the initial image to obtain the spatial distribution information of the key points of the target face; then, a 3D facial model is constructed based on the facial key points in 3D space; then, based on the 3D facial model and the light source parameter information of the simulated lighting source, target illumination spatial distribution information (such as the first illumination spatial distribution information corresponding to specular reflection and the second illumination spatial distribution information corresponding to diffuse reflection) is generated; finally, image fusion is performed between the target illumination spatial distribution information and the initial image to obtain the target illuminated image. Based on this, the technical effects produced by the embodiments provided in this application include at least the following points:

[0333] On the one hand, by using a preset lighting model, the 3D face model constructed based on facial key points is simulated with lighting, and the intensity of reflected light is calculated. Then, the spatial distribution information of target lighting is generated. This can simulate the lighting direction and intensity corresponding to each unit surface in the 3D face model, making the lighting effect added to the initial image closer to the real lighting effect, thereby ensuring the authenticity of the target lighting image shown to the user.

[0334] On the other hand, in the process of constructing a 3D face model, the location information of the facial key points obtained by using the neural network model includes not only the planar coordinate information of the facial key points, but also the depth information of the facial key points. This allows us to obtain the spatial distribution information of the key points of the target face. Based on this spatial distribution information, we can directly establish the connection relationship between the facial key points to obtain a 3D face model, thereby improving the construction efficiency of the 3D face model.

[0335] On the other hand, in the process of generating illumination spatial distribution information, since the face stereo model is obtained by establishing the connection relationship between the key points of the face using the neighborhood growth method, the face stereo model can include multiple unit surfaces. Each unit surface is used as the smallest computing unit to calculate the reflected light intensity corresponding to each unit surface in the face stereo model. Then, the pixel value of each pixel point on the unit surface is determined. Then, the target illumination spatial distribution information is generated based on the pixel value. This not only ensures the accuracy of the calculation of reflected light intensity, but also improves the generation efficiency of illumination spatial distribution information.

[0336] In addition, during the image fusion process, considering that the initial image already contains the luminous effects brought about by ambient light and self-emission, the four types of lighting conditions—ambient light, self-emission, specular reflection, and diffuse reflection—are distinguished. Only the spatial distribution information of the lighting corresponding to specular reflection and the spatial distribution information of the lighting corresponding to diffuse reflection are generated. This ensures that the pixel value of each pixel in the target lighting spatial distribution information is only related to the intensity of specular reflection and diffuse reflection, and is independent of the intensity of the lighting generated by ambient light and self-emission. This avoids the problem of oversaturation of the lighting in the target image due to the repeated superposition of the luminous effects brought about by ambient light and self-emission.

[0337] Scene 5

[0338] In this scenario, where the image lighting type is subject lighting and the target subject is a human face, in one case, considering that the initial image may contain not only the image area of ​​the target face but also the image area of ​​the corresponding human body, to ensure the coordination of the lighting effects on the target face and the target human body, let's take an initial image containing a target face and its corresponding human body as an example. Figure 24 As shown, the specific implementation process for generating the target lighting image corresponding to the initial image may include:

[0339] The image segmentation module performs region segmentation on the initial image to obtain a first sub-image containing the target face and a second sub-image containing the target body. For example, a pre-trained image segmentation model can be used to segment the initial image into image regions containing the face and body, resulting in an image region containing the target face (i.e., the first sub-image) and an image region containing the target body (i.e., the second sub-image).

[0340] The key point extraction module performs key point extraction processing on the first sub-image to obtain the spatial distribution information of key points on the target face. The process for determining the spatial distribution information of key points can be found above. Figure 20 The specific implementation process will not be elaborated here.

[0341] The 3D model construction module performs 3D reconstruction based on the spatial distribution information of key points on the target face to obtain a 3D model of the target face. The process of constructing the 3D face model can be found above. Figure 20 The specific implementation process will not be elaborated here.

[0342] The illumination distribution determination module, based on the light source parameter information of the simulated lighting source and the aforementioned 3D face model, generates first illumination spatial distribution information corresponding to the specular reflection of the target face and second illumination spatial distribution information corresponding to the diffuse reflection. The process for determining the first and second illumination spatial distribution information can be found above. Figure 20 The specific implementation process will not be elaborated here.

[0343] The illumination distribution determination module performs planar projection based on the first illumination spatial distribution information to obtain the first illumination plane distribution information corresponding to the target face. Specifically, it establishes a correspondence between each first pixel value in the first illumination spatial distribution information and each pixel in the first sub-image to obtain the first illumination plane distribution information.

[0344] The illumination distribution determination module performs planar projection based on the second illumination spatial distribution information to obtain the second illumination plane distribution information corresponding to the target face. Specifically, it establishes a correspondence between each second pixel value in the second illumination spatial distribution information and each pixel in the first sub-image to obtain the second illumination plane distribution information.

[0345] The illumination distribution prediction module predicts the human body illumination distribution based on the second sub-image, the first illumination plane distribution information, and the light source parameter information, to obtain the third illumination plane distribution information of the target human body. The third illumination plane distribution information includes the third pixel value of each pixel in the second sub-image, with each third pixel value representing the specular reflection color of a specific pixel on the target human body.

[0346] For example, using a pre-trained first illumination distribution prediction model, the human body illumination distribution is predicted based on the relative positional relationship between the second and first sub-images and the first illumination plane distribution information, to obtain the third illumination plane distribution information of the target human body. The first illumination distribution prediction model is used to predict the specular reflection illumination distribution of the target human body based on the specular reflection illumination distribution of the target face. Furthermore, during the human body illumination distribution prediction process, light source parameter information can also be used as a reference, and the coordinate information of each pixel in the initial image can characterize the relative positional relationship between the target human body and the target face. Therefore, the first illumination plane distribution information, light source parameter information, and the coordinate information of each pixel in the initial image can be input into the first illumination distribution prediction model for illumination prediction to obtain the third illumination plane distribution information of the target human body.

[0347] The illumination distribution prediction module predicts the human body illumination distribution based on the second sub-image, the second illumination plane distribution information, and the light source parameter information, thereby obtaining the fourth illumination plane distribution information of the target human body. The fourth illumination plane distribution information includes the fourth pixel value of each pixel in the second sub-image, with each fourth pixel value representing the diffuse reflection light color of a specific pixel on the target human body.

[0348] For example, using a pre-trained second illumination distribution prediction model, the human body illumination distribution is predicted based on the relative positional relationship between the second sub-image and the first sub-image, and the second illumination plane distribution information, to obtain the fourth illumination plane distribution information of the target human body. The second illumination distribution prediction model is used to predict the diffuse illumination distribution of the target human body based on the diffuse illumination distribution of the target face. Furthermore, during the human body illumination distribution prediction process, light source parameter information can also be used as a reference, and the coordinate information of each pixel in the initial image can characterize the relative positional relationship between the target human body and the target face. Therefore, the second illumination plane distribution information, light source parameter information, and the coordinate information of each pixel in the initial image can be input into the second illumination distribution prediction model for illumination prediction to obtain the fourth illumination plane distribution information of the target human body.

[0349] The illumination distribution combination module performs portrait synthesis processing based on the aforementioned first and third illumination plane distribution information to obtain first overall illumination distribution information. The first illumination plane distribution information characterizes the highlight reflection illumination distribution of the target face, while the third illumination plane information characterizes the highlight reflection illumination distribution of the target body. Therefore, based on the relative positional relationship between the target face and the target body in the initial image, the first and third illumination plane distribution information are stitched together to obtain the first overall illumination distribution information. This first overall illumination distribution information characterizes the highlight reflection illumination distribution of the target user's portrait.

[0350] The illumination distribution combination module performs portrait synthesis processing based on the aforementioned second and fourth illumination plane distribution information to obtain second overall illumination distribution information. The second illumination plane distribution information characterizes the diffuse illumination distribution of the target face, while the fourth illumination plane distribution information characterizes the diffuse illumination distribution of the target body. Therefore, based on the relative positional relationship between the target face and the target body in the initial image, the second and fourth illumination plane distribution information are stitched together to obtain the second overall illumination distribution information. This second overall illumination distribution information characterizes the diffuse illumination distribution of the target user's portrait.

[0351] The image fusion module performs image fusion processing based on the initial image, the aforementioned first overall illumination distribution information, and the second overall illumination distribution information to obtain the target lighting image. Specifically, firstly, based on the first overall illumination distribution information, the first pixel value of the target pixel and the third pixel value of the human pixel in the initial image are determined; and secondly, based on the second overall illumination distribution information, the second pixel value of the target pixel and the fourth pixel value of the human pixel in the initial image are determined; the target pixel is any pixel on the 3D face model, and the human pixel is any pixel in the image region where the target human body is located; then, the original pixel value, the first pixel value, and the second pixel value of the target pixel are weighted and summed to obtain the fused pixel value of the target pixel; and the original pixel value, the third pixel value, and the fourth pixel value of the human pixel are weighted and summed to obtain the fused pixel value of the human pixel; wherein, the original pixel value is used to characterize the pixel color of the initial image; next, based on the original pixel values ​​of each non-target pixel in the initial image, the fused pixel values ​​of each target pixel, and the fused pixel values ​​of each human pixel, the target lighting image is generated; the non-target pixels are the pixels in the initial image that have not undergone lighting processing.

[0352] For images where the lighting type is subject-based lighting, another scenario considers that the initial image may include not only the image area of ​​the target face but also the image area of ​​the background subject. Taking an initial image containing both a target face and a background subject as an example... Figure 25 As shown, the specific implementation process for generating the target lighting image corresponding to the initial image may include:

[0353] The image segmentation module performs region segmentation on the initial image to obtain a third sub-image containing the target face and a fourth sub-image containing the background subject. For example, a pre-trained image segmentation model can be used to segment the initial image into image regions containing the face and background, resulting in an image region containing the target face (i.e., the third sub-image) and an image region containing the background subject (i.e., the fourth sub-image).

[0354] The key point extraction module performs key point extraction processing on the aforementioned third sub-image to obtain the spatial distribution information of key points on the target face. The process for determining the spatial distribution information of key points can be found above. Figure 20 The specific implementation process will not be elaborated here.

[0355] The 3D model construction module performs 3D reconstruction based on the spatial distribution information of key points on the target face to obtain a 3D model of the target face. The process of constructing the 3D face model can be found above. Figure 20 The specific implementation process will not be elaborated here.

[0356] The illumination distribution determination module, based on the light source parameter information of the simulated lighting source and the aforementioned 3D face model, generates first illumination spatial distribution information corresponding to the specular reflection of the target face and second illumination spatial distribution information corresponding to the diffuse reflection. The process for determining the first and second illumination spatial distribution information can be found above. Figure 20 The specific implementation process will not be elaborated here.

[0357] The illumination distribution determination module performs planar projection based on the first illumination spatial distribution information to obtain the first illumination plane distribution information corresponding to the target face. Specifically, it establishes a correspondence between each first pixel value in the first illumination spatial distribution information and each pixel in the third sub-image to obtain the first illumination plane distribution information.

[0358] The illumination distribution determination module performs planar projection based on the second illumination spatial distribution information to obtain the second illumination plane distribution information corresponding to the target face. Specifically, it establishes a correspondence between each second pixel value in the second illumination spatial distribution information and each pixel in the third sub-image to obtain the second illumination plane distribution information.

[0359] The illumination distribution prediction module predicts the background illumination distribution based on the aforementioned fourth sub-image, the first illumination plane distribution information, and the light source parameter information, to obtain the fifth illumination plane distribution information of the background object. The fifth illumination plane distribution information includes the fifth pixel value of each pixel in the fourth sub-image, with each fifth pixel value representing the specular reflection color of a specific pixel in the background object.

[0360] For example, using a pre-trained third illumination distribution prediction model, background illumination distribution prediction is performed based on the relative positional relationship between the fourth and third sub-images, the first illumination plane distribution information, and the light source parameter information, to obtain the fifth illumination plane distribution information of the background object. The third illumination distribution prediction model is used to predict the highlight reflection illumination distribution of the background object based on the highlight reflection illumination distribution of the target face; wherein, during the model training phase, the training sample sets used by the third illumination distribution prediction model and the first illumination distribution prediction model can be different. In addition, during the background illumination distribution prediction process, the light source parameter information can also be used as a reference, and the coordinate information of each pixel in the initial image can characterize the relative positional relationship between the background object and the target face. Therefore, the first illumination plane distribution information, the light source parameter information, and the coordinate information of each pixel in the initial image can be input into the third illumination distribution prediction model for illumination prediction to obtain the fifth illumination plane distribution information of the background object.

[0361] The illumination distribution prediction module predicts the background illumination distribution based on the aforementioned fourth sub-image, the second illumination plane distribution information, and the light source parameter information, to obtain the sixth illumination plane distribution information of the background object. The sixth illumination plane distribution information includes the sixth pixel value of each pixel in the fourth sub-image, with each sixth pixel value representing the diffuse reflection color of a specific pixel in the background object.

[0362] For example, using a pre-trained fourth illumination distribution prediction model, background illumination distribution prediction is performed based on the relative positional relationship between the fourth and third sub-images and the second illumination plane distribution information to obtain the sixth illumination plane distribution information of the background object. The fourth illumination distribution prediction model is used to predict the diffuse illumination distribution of the background object based on the diffuse illumination distribution of the target face; wherein, during the model training phase, the training sample sets used by the fourth illumination distribution prediction model and the second illumination distribution prediction model can be different. In addition, during the background illumination distribution prediction process, light source parameter information can also be used as a reference, and the coordinate information of each pixel in the initial image can represent the relative positional relationship between the background object and the target face. Therefore, the second illumination plane distribution information, light source parameter information, and the coordinate information of each pixel in the initial image can be input into the fourth illumination distribution prediction model for illumination prediction to obtain the sixth illumination plane distribution information of the background object.

[0363] The illumination distribution combination module performs foreground and background compositing based on the aforementioned first and fifth illumination plane distribution information to obtain first panoramic illumination distribution information. The first illumination plane distribution information characterizes the highlight reflection illumination distribution of the target face, while the fifth illumination plane information characterizes the highlight reflection illumination distribution of the background objects. Therefore, based on the relative positional relationship between the target face and the background objects in the initial image, the first and fifth illumination plane distribution information are stitched together to obtain the first panoramic illumination distribution information. This first panoramic illumination distribution information characterizes the highlight reflection illumination distribution of the entire panorama.

[0364] The illumination distribution combination module performs foreground and background compositing based on the aforementioned second and sixth illumination plane distribution information to obtain second panoramic illumination distribution information. The second illumination plane distribution information characterizes the diffuse illumination distribution of the target face, while the sixth illumination plane distribution information characterizes the diffuse illumination distribution of the background objects. Therefore, based on the relative positional relationship between the target face and the background objects in the initial image, the second and sixth illumination plane distribution information are stitched together to obtain the second panoramic illumination distribution information. This second panoramic illumination distribution information characterizes the diffuse illumination distribution of the entire panorama.

[0365] The image fusion module performs image fusion processing based on the initial image, the aforementioned first panoramic illumination distribution information, and the second panoramic illumination distribution information to obtain the target lighting image. Specifically, firstly, based on the first panoramic illumination distribution information, the first pixel value of the target pixel and the fifth pixel value of the background pixel in the initial image are determined; and secondly, based on the second panoramic illumination distribution information, the second pixel value of the target pixel and the sixth pixel value of the background pixel in the initial image are determined; the target pixel is any pixel on the 3D face model, and the background pixel is any pixel in the image area where the background object is located; then, the original pixel value, the first pixel value, and the second pixel value of the target pixel are weighted and summed to obtain the fused pixel value of the target pixel; and the original pixel value, the fifth pixel value, and the sixth pixel value of the background pixel are weighted and summed to obtain the fused pixel value of the background pixel; wherein, the original pixel value is used to characterize the pixel color of the initial image; next, based on the original pixel values ​​of each non-target pixel in the initial image, the fused pixel values ​​of each target pixel, and the fused pixel values ​​of each background pixel, a target lighting image is generated; the non-target pixels are the pixels in the initial image that have not been lit.

[0366] Understandably, if the initial image includes a target face, the target human body corresponding to the target face, and the background shooting object, then the process of generating the target lighting image can refer to the specific implementation process given in scenarios four and five above, and will not be repeated here.

[0367] Furthermore, if the initial image contains two or more target faces, multiple target faces can be segmented from the initial image first, and the spatial distribution information of key points corresponding to each target face can be determined. Then, for each target face, a corresponding 3D face model can be constructed. Next, for each 3D face model, corresponding illumination spatial distribution information can be generated. Finally, image fusion processing can be performed based on the initial image and the illumination spatial distribution information corresponding to each target face to obtain the target lighting image. The specific implementation process can be referred to the above embodiment, and will not be repeated here.

[0368] It should be noted that the image processing method provided in this application embodiment is applicable to lighting processing of video data; for example, lighting processing can be performed on some or all image frames in the video data to obtain the video data after lighting processing; wherein, the specific implementation process of lighting processing for each image frame can be referred to the above detailed description, and will not be repeated here.

[0369] This embodiment also provides an electronic device, which includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by one or more processors, the electronic device performs the above-described related method steps to implement the image processing method in the above embodiment.

[0370] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0371] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0372] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0373] In addition, embodiments of this application also provide a chip, which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuits communicate with each other through internal connection paths, and the processing circuits execute the above-described related method steps to implement the image processing method in the above embodiments, so as to control the receiving pins to receive signals and control the transmitting pins to transmit signals.

[0374] In this embodiment, the electronic devices (such as mobile phones), computer storage media, computer program products, devices or chips are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0375] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0376] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0377] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, including: The first preview interface of the target application is displayed; the target application has an image acquisition function, and the first preview interface includes a first control. In response to a first operation on the first control, a second preview interface of the target application is displayed; the second preview interface includes a second preview image, which is obtained by lighting the background area of ​​the target object in the first preview image, and the first preview image is a frame preview image captured by the electronic device.

2. The method according to claim 1, characterized in that, Before displaying the second preview interface of the target application, the following is also included: Displays a third preview interface of the target application; the third preview interface includes a second control; In response to a second operation on the second control, at least one target object is determined from a plurality of captured objects in the first preview image; the background area is determined based on the contour information of the at least one target object.

3. The method according to claim 2, characterized in that, Each of the second controls corresponds to an object identifier of a shooting object, wherein the target object is the shooting object corresponding to the object identifier selected by the user.

4. The method according to claim 1, characterized in that, Before displaying the second preview interface of the target application, the following is also included: Display a first settings interface, and in response to a third operation performed on the first settings interface, determine the first parameter information; The background area of ​​the target object in the first preview image is illuminated based on the first parameter information to obtain the second preview image.

5. The method according to claim 4, characterized in that, The second preview interface also includes a third control; After displaying the second preview interface of the target application, the following is also included: In response to a fourth operation on the third control, a first captured image is acquired, and the background area of ​​the target object in the first captured image is illuminated according to the first parameter information to obtain a second captured image; Store the second captured image.

6. The method according to claim 4, characterized in that, The second preview interface also includes a fourth control; After displaying the second preview interface of the target application, the following is also included: In response to a fifth operation on the fourth control, the second parameter information is determined; The background area of ​​the target object in the third preview image is illuminated according to the second parameter information to obtain the fourth preview image; the third preview image is a frame preview image captured by the electronic device. The fourth preview image is displayed in the second preview interface.

7. The method according to claim 6, characterized in that, The determination of the second parameter information includes: Displaying a human-computer interaction interface; the human-computer interaction interface includes a fifth control; In response to a sixth operation on the fifth control, user input information is obtained; the user input information is used to describe the expected lighting effect. Based on the user input information, the light source parameters are matched to determine the second parameter information.

8. The method according to claim 1, characterized in that, Also includes: The first interface of the gallery application is displayed; the first interface includes a first image and a sixth control, wherein the first image is the image selected by the user; In response to a seventh operation on the sixth control, a second settings interface is displayed; In response to the eighth operation performed on the second settings interface, the third parameter information is determined; The background area of ​​the target object in the first image is illuminated according to the third parameter information to obtain a second image, which is then displayed on the second interface of the image library application.

9. The method according to claim 4, characterized in that, The step of applying lighting to the background area of ​​the target object in the first preview image based on the first parameter information to obtain the second preview image includes: Perform subject segmentation processing on the first preview image to obtain the first contour information of the target object in the first preview image; Based on the first parameter information, the target lighting template is determined from the lighting template set; The background region and the subject edge region are determined based on the first contour information, and the illumination coefficient distribution information of the first preview image is determined based on the illumination coefficient of the background region and the illumination coefficient of the subject edge region. Based on the target lighting template and the lighting coefficient distribution information, a lighting weighted rendering map is obtained; The second preview image is obtained by performing image fusion processing based on the first preview image and the light-weighted rendering map.

10. The method according to claim 9, characterized in that, The first parameter information includes the direction of the light source; Determining the background region and the subject edge region based on the first contour information includes: If the light source direction is a forward or backlight direction, then based on the first contour information, the second contour information of the target object is determined, and based on the first contour information and the second contour information, the background area and the subject edge area are determined; the first contour information is used to characterize the original contour position of the target object, and the second contour information is used to characterize the inner contour position of the target object; If the light source direction is a side-backlight direction, then the third contour information of the target object is determined based on the first contour information and the light source direction, and the background area and the subject edge area are determined based on the first contour information and the third contour information; the third contour information is used to characterize the projected contour position of the target object along the light source direction.

11. The method according to claim 9, characterized in that, Before determining the illumination coefficient distribution information of the first preview image, the process also includes: According to the preset region division rules, multiple sub-regions are determined in the main body edge region; each sub-region corresponds to a partition type; The illumination coefficient of the main body edge region is determined based on the illumination coefficient corresponding to the partition type of each sub-region.

12. The method according to claim 9, characterized in that, The step of performing image fusion processing based on the first preview image and the illumination-weighted rendering map to obtain the second preview image includes: The color gamut of the first preview image is adjusted to obtain the image to be illuminated; The second preview image is obtained by performing image fusion processing based on the image to be lit and the light weighted rendering image.

13. The method according to claim 9, characterized in that, The method further includes: Acquire multiple first illumination templates; the multiple first illumination templates include a real-shot illumination distribution template and a simulated illumination distribution template; A second lighting template is obtained by performing a lighting color conversion process on any of the first lighting templates. The lighting template set is generated based on a plurality of first lighting templates and a plurality of second lighting templates.

14. The method according to claim 1, characterized in that, The method further includes: Displays a fourth preview interface of the target application; the fourth preview interface includes a seventh control; In response to the ninth operation on the seventh control, the fifth preview interface of the target application is displayed; the fifth preview interface includes a sixth preview image, which is obtained by illuminating the target object in the fifth preview image with a simulated light source, and the fifth preview image is a frame preview image captured by the electronic device.

15. The method according to claim 14, characterized in that, Before displaying the fifth preview interface of the target application, the following is also included: Display a third setting interface, and in response to a tenth operation performed on the third setting interface, determine the fourth parameter information of the simulated light source; The target object in the fifth preview image is illuminated according to the fourth parameter information to obtain the sixth preview image.

16. The method according to claim 15, characterized in that, The step of applying lighting processing to the target object in the fifth preview image based on the fourth parameter information to obtain the sixth preview image includes: The key point extraction process is performed on the fifth preview image to obtain the key point distribution information of the target face in the fifth preview image; Based on the key point distribution information, a three-dimensional reconstruction process is performed to obtain a three-dimensional model of the target face; Based on the fourth parameter information, the three-dimensional model is simulated with lighting to obtain the target illumination distribution information corresponding to the three-dimensional model. The sixth preview image is obtained by performing image fusion processing based on the fifth preview image and the target illumination distribution information.

17. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the image processing method as described in any one of claims 1 to 16.

18. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, the electronic device causes the electronic device to perform the image processing method as described in any one of claims 1 to 16.

19. A chip used in an electronic device, the chip comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 16.

Images