Image processing method, electronic equipment and storage medium

Abstract

The invention provides an image processing method, electronic equipment and a storage medium. The method comprises the following steps: in a scene of performing image acquisition by using a target application, triggering to perform lighting processing on a to-be-displayed first preview image through a first control on a photographing preview interface to obtain a second preview image; and then displaying the second preview image on a photographing preview interface. If the trigger operation on the first control is detected, the collected preview image is not directly displayed, the preview image to be displayed is firstly subjected to lighting processing, and then the preview image subjected to lighting processing is displayed on the photographing preview interface. In this way, it can be achieved that the lighting effect is added to the image area where the target object in the preview image is located based on the simulation light source in the photographing preview process, so that a user can determine whether the image lighting effect meets the expectation or not through the preview image displayed on the photographing preview interface, and therefore the image lighting processing process is preposed; furthermore, only the shot image after the lighting processing can be stored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent terminals, and in particular to an image processing method, an electronic device, and a storage medium. BACKGROUND

[0002] With the continuous upgrading of the functions of electronic devices, the application of cameras in electronic devices is becoming more and more widespread to meet the needs of users for taking pictures. On the one hand, for daily picture-taking scenarios, there may be a need for light processing of images due to poor display effects of pictures taken in relatively dark light. On the other hand, for application scenarios in which a model is trained based on sample images, there may be a need for images with different light effects as sample images to ensure the diversification of the light effects of sample images. At this time, there is also a need for light processing of images. SUMMARY

[0003] To solve the above technical problems, the present application provides an image processing method, an electronic device, and a storage medium. In the method, the addition of a light effect based on a simulated light source to an image region in which a target object is located in a preview image during a picture-taking preview process can be triggered, and the preview image after light processing is displayed on a picture-taking preview interface, so that a user can determine whether the light effect of the image meets expectations through the preview image displayed on the picture-taking preview interface, thereby realizing the front positioning of the light processing process of the image, and then selectively storing only the captured image after light processing in the case where the light effect of the preview image displayed on the picture-taking preview interface meets expectations.

[0004] In a first aspect, an image processing method is provided. The method is applied to an electronic device and includes: displaying a first preview interface of a target application; the target application has an image capturing function, and the first preview interface includes a first control; in response to a first operation on the first control, a second preview interface of the target application is displayed; the second preview interface includes a second preview image, and the second preview image is obtained by performing light processing on a target object in a first preview image based on a simulated light source, and the first preview image is a frame of preview image captured by the electronic device.

[0005] For example, the electronic device can be a mobile phone, a notebook computer, a tablet computer, or the like. The first interface can be a picture-taking preview interface; the target application can be a camera application or any third-party application with a picture-taking function, such as a chat application or a short video application. The first control can be a light control mentioned in the embodiments below; the simulated light source can be a simulated light source mentioned in the embodiments below, which refers to a light source created in a digital environment to simulate the characteristics of a light source in the real world and simulate the lighting effect of a real light source.

[0006] Exemplarily, the process of lighting the first preview image can include key point extraction on the target object in the first preview image, construction of a three-dimensional model of the target object, determination of lighting distribution information of the three-dimensional model, and pixel value weighting calculation processing on the first preview image and the lighting distribution information.

[0007] In addition, for the process of image lighting, a lighting effect can be added to an image region where the target object is located in the first preview image. The target object can be a target face or other shooting objects, such as an animal head, and the like.

[0008] Specifically, for the process of image lighting, the lighting distribution information of the target object is simulated based on the simulated light source, and then the target object in the first preview image is lighted based on the lighting distribution information, thereby improving the authenticity of the image lighting effect. Optionally, to further improve the authenticity of the image lighting effect, a three-dimensional model of the target object can be constructed first, and the lighting spatial distribution information of the target object is simulated based on the simulated light source and the three-dimensional model, and then the target object in the first preview image is lighted based on the lighting spatial distribution information.

[0009] In this way, in the scene of image acquisition using the target application, the first control on the photograph preview interface can be used to trigger the lighting of the first preview image to be displayed on the photograph preview interface based on the simulated light source, and the second preview image is obtained; and then the second preview image is displayed on the photograph preview interface. That is, by continuously collecting preview images through the camera, the preview image displayed on the photograph preview interface is also refreshed in real time, and multiple preview images are displayed one by one. If the triggering operation on the first control is detected, the next frame of preview image to be displayed will not be directly displayed on the photograph preview interface, but the preview image to be displayed is lighted first, and then the lighted preview image is displayed on the photograph preview interface. Based on this, the process of adding a lighting effect to the image region where the target object is located in the preview image based on the simulated light source can be triggered during the photograph preview process, and the lighted preview image is displayed on the photograph preview interface, so that the user can determine whether the image lighting effect meets the expectation through the preview image displayed on the photograph preview interface, thereby realizing the prepositioning of the process of image lighting, and then selectively storing only the lighted shooting image when the lighting effect of the preview image displayed on the photograph preview interface meets the expectation.

[0010] According to the first aspect, in the method, before displaying the second preview interface of the target application, the first setting interface is displayed, and the first parameter information of the simulated light source is determined in response to the second operation performed on the first setting interface; and the target object in the first preview image is lighted according to the first parameter information, and the second preview image is obtained.

[0011] For example, the first setting interface includes at least one parameter setting control, each parameter setting control being configured to set a lighting parameter; the first setting interface can be a lighting parameter setting interface mentioned in the embodiments below; the parameter setting control can be a control configured to set a value of a lighting parameter; the first parameter information can be light source parameter information mentioned in the embodiments below, which can be a default value, a parameter value set by a user, or a parameter value determined automatically based on user input information; the first parameter information includes parameter values corresponding to a plurality of lighting parameters respectively, and the lighting parameters can be light source parameters of a simulated light source.

[0012] In this way, after detecting the triggering operation on the lighting control, a lighting parameter setting interface is provided for the user, and the user can set the value of the image lighting parameter according to actual needs, thereby meeting the personalized needs of the user for the image lighting effect.

[0013] According to the first aspect, or any one of the implementations of the first aspect, in the method, the second preview interface further includes a second control; after displaying the second preview interface of the target application, the method further includes: in response to a third operation on the second control, capturing a first photographed image, and performing lighting processing on the target object in the first photographed image according to the first parameter information to obtain a second photographed image; and storing the second photographed image.

[0014] The second control can be a photographing control. For a photographing scenario, if it is detected that the user performs a click operation on the lighting control, the preview image needs to be processed before the user clicks the photographing control, and the target lighting image corresponding to the preview image (i.e., the second preview image) is displayed on the photographing preview interface; after the user clicks the photographing control, the photographed image needs to be processed, and the target lighting image corresponding to the photographed image (i.e., the second photographed image) is stored. The process of processing the photographed image can refer to the process of processing the first preview image, which will not be described herein again.

[0015] In this way, if the lighting effect of the target lighting image (i.e., the second preview image) displayed on the photographing preview interface meets the user's expectation, the user can trigger the camera to capture the photographed image by clicking the photographing control; then, the photographed image is processed first to obtain the target lighting image corresponding to the photographed image (i.e., the second photographed image), and the target lighting image is stored, so that the photographed image after lighting processing can be viewed in the gallery application; at this time, the photographed image without lighting processing can not be stored, but the target lighting image with the lighting effect meeting the expectation is directly stored, thereby eliminating the step of deleting the photographed image without lighting processing to release the memory.

[0016] In the method according to the first aspect, or any possible implementation mode of the first aspect, the second preview interface further includes a third control; and the method further includes: in response to a fourth operation on the third control, determining second parameter information of the simulated light source; performing light processing on the target object in the third preview image according to the second parameter information to obtain a fourth preview image; the third preview image is a preview image captured by the electronic device; and displaying the fourth preview image on the second preview interface.

[0017] For example, the second parameter information can be adjusted light source parameter information; the third control can be a control for triggering re-lighting, and the third control can be the same as or different from the first control. For example, if the third control is the same as the first control, the second parameter information can be determined in response to an adjusted parameter value set by the user on the light parameter setting interface; for another example, if the third control is different from the first control, the third control can be a newly added human-computer interaction control, and correspondingly, the second parameter information can be obtained by matching parameter values based on description information of an expected light effect input by the user; wherein the third preview image is captured later than the first preview image.

[0018] In this way, if the light effect of the target light image displayed on the photograph preview interface does not meet the user's expectation, the user can trigger adjustment of the first parameter information through the third control to obtain the second parameter information, and then perform light processing on the preview image using the second parameter information to obtain a new target light image, until the light effect of the target light image meets the user's expectation, so that timely adjustment of the light effect of the image can be triggered during the photograph preview process to ensure that the image stored in the gallery is the target light image that meets the user's expectation, thereby eliminating the user's deletion operation on the lighted image in the gallery application and the step of triggering re-lighting of the image. In addition, for the case where the user triggers re-lighting of the image, it can be because the user does not understand the correspondence between the light parameter value and the light effect, and thus cannot determine the accuracy of the set light parameter. Based on this, by adding a third control different from the first control, after detecting the triggering operation of the user on the third control, a human-computer interaction interface is displayed, and description information of an expected light effect is input through an information input control on the human-computer interaction interface, so that the adjusted light source parameter information can be obtained by matching parameter values based on the description information of the expected light effect input by the user. Therefore, for the user who does not understand the correspondence between the light parameter value and the light effect, the user only needs to describe the expected light effect, without needing to pay attention to the specific value of each light parameter.

[0019] Further, after displaying the fourth preview image on the second preview interface, the method further includes: in response to a triggering operation on a photographing control on the second preview interface, collecting a third photographing image, performing light processing on a target object in the third photographing image according to the second parameter information to obtain a fourth photographing image, and storing the fourth photographing image. In this way, if the light effect of the target light image (i.e., the fourth preview image) displayed on the photographing preview interface after the light parameter is adjusted meets the user's expectation, the user can trigger the collection of the photographing image by the camera by clicking the photographing control. Then, the photographing image is first light processed to obtain the target light image (i.e., the fourth photographing image) corresponding to the photographing image, and the target light image is stored. At this time, the photographing image without light processing can not be stored, but the target light image with the light effect meeting the expectation is directly stored, thereby saving the step of deleting the photographing image without light processing to release the memory.

[0020] According to the first aspect, or any one of the implementations of the first aspect, in the method, the method further includes: displaying a first interface of a gallery application; the first interface includes a first image and a fourth control, the first image being an image selected by a user; in response to a fifth operation on the fourth control, displaying a second setting interface; in response to a sixth operation performed on the second setting interface, determining third parameter information; performing light processing on a target object in the first image according to the third parameter information to obtain a second image, and displaying the second image on a second interface of the gallery application.

[0021] For example, the first interface can be a photo display interface of the gallery application, the first image can be a gallery image to be light processed mentioned in the embodiments below, the fourth control can be a light control similar to the first control described above, and the second image can be a third light image mentioned in the embodiments below, and the second interface can be an image editing interface of the gallery application. The first image can be an image without light processing or an image with light processing.

[0022] For example, for the first image being an image with light processing, the first image can be first restored to obtain a corresponding initial image, and then the initial image is light processed to obtain the second image. If the corresponding initial image is stored during the light processing of the first image, the corresponding initial image of the first image can be directly obtained from the memory, and then the initial image is light processed to obtain the second image. The present application does not limit this.

[0023] In addition, for the process of performing light processing on the image in the gallery application, a light setting interface (i.e., a second setting interface) can also be provided for the user, and the implementation process can refer to the above-mentioned process of setting the first parameter information, which will not be repeated here. In addition, the process of performing light processing on the first image can refer to the above-mentioned process of performing light processing on the first preview image, which will not be repeated here.

[0024] In this way, the user can not only trigger the light processing on the target object in the preview image based on the simulated light source during the preview process, but also trigger the light processing on the target object in any image in the gallery application based on the simulated light source, thereby ensuring the flexibility of the user triggering the light processing on the image. For the scenario of performing light processing on the image in the gallery application, for example, if the user is not satisfied with the light effect of the stored target light image, the user can trigger the re-light processing on the image. For another example, the user can trigger the light processing on the image stored in the gallery from other devices.

[0025] In addition, if the light effect of the target light image displayed on the second interface meets the user's expectation, the user can trigger the saving of the target light image corresponding to the image selected by the user through the saving control, so as to view the target light image at any time from the gallery application.

[0026] According to the first aspect, or any one of the implementation manners of the first aspect, in the method, the light processing on the target object in the first preview image according to the first parameter information to obtain the second preview image can include: performing key point extraction processing on the first preview image to obtain key point distribution information of the target object in the first preview image; performing three-dimensional reconstruction processing according to the key point distribution information to obtain a three-dimensional model of the target object; performing simulated light processing on the three-dimensional model according to the first parameter information to obtain target light distribution information corresponding to the three-dimensional model; and performing image fusion processing according to the first preview image and the target light distribution information to obtain the second preview image.

[0027] Exemplarily, the first preview image can be the initial image mentioned in the embodiments below, and the second preview image can be the target lighted image mentioned in the embodiments below. The key point distribution information can be the key point spatial distribution information mentioned in the embodiments below, and the key point spatial distribution information includes spatial position information of the key points of the target object. The target light distribution information can be the target light spatial distribution information mentioned in the embodiments below, and the target light spatial distribution information reflects the light distribution in the three-dimensional space, and therefore, can also be referred to as a three-dimensional spatial distribution diagram of light intensity. The process of the image fusion processing can mainly include projecting each pixel point in the target light spatial distribution information to a two-dimensional plane distribution diagram, and then performing weighted summation on the pixel value of each pixel point in the two-dimensional plane distribution diagram and the pixel value of the corresponding pixel point in the first preview image to obtain a fusion pixel value. Further, the target lighted image is generated based on the fusion pixel value of each pixel point.

[0028] In this way, for the process of performing lighted processing on the initial image, if the target object to which the lighted effect needs to be added is a target face, the face key points are first extracted from the initial image, then the face three-dimensional model is constructed based on the face key points in the three-dimensional space, the target light spatial distribution information (at least including the light spatial distribution information corresponding to the highlight reflection and the light spatial distribution information corresponding to the diffuse reflection) of the target face is generated based on the face three-dimensional model and the light source parameter information of the simulated light source, and the image fusion processing is performed on the target light spatial distribution information and the initial image to obtain the target lighted image. In this way, the light direction and the light intensity corresponding to each unit surface in the face three-dimensional model can be simulated, so that the lighted effect added to the initial image is closer to the real lighted effect, and the authenticity of the target lighted image displayed to the user is further ensured.

[0029] According to the first aspect, or any one of the implementations of the first aspect, in the method, the three-dimensional reconstruction processing is performed according to the key point distribution information to obtain the three-dimensional model of the target object, which can include: selecting a first point and a second point from the plurality of key points in the key point distribution information; determining a current growth point from the plurality of key points in turn by using a region growing method, starting from the first point and ending at the second point; and establishing a connection relationship between the current growth point and the region growth point of the current growth point to obtain the three-dimensional model of the target object.

[0030] Exemplarily, the first point can be the starting growth point mentioned in the embodiments below, and the second point can be the termination growth point mentioned in the embodiments below.

[0031] In this way, since the face three-dimensional model is obtained by using the region growing method to establish the connection relationship between the face key points, the face three-dimensional model can include multiple unit faces, each unit face is taken as a minimum calculation unit, the corresponding reflected light intensity of each unit face in the face three-dimensional model is calculated, then the pixel value of each pixel point on the unit face is determined, and then the target light space distribution information is generated based on the pixel value, so that the calculation accuracy of the reflected light intensity can be ensured, and the generation efficiency of the light space distribution information can be improved.

[0032] According to the first aspect or any one of the implementations of the first aspect, in the method, the simulating lighting processing on the three-dimensional model according to the first parameter information to obtain the target light distribution information corresponding to the three-dimensional model can include: determining target parameter information of each unit face in the three-dimensional model according to the first parameter information; performing high-light reflection light calculation according to the target parameter information to obtain first light distribution information; performing diffuse reflection light calculation according to the target parameter information to obtain second light distribution information; and determining the first light distribution information and the second light distribution information as the target light distribution information.

[0033] For example, the target parameter information can be the target light parameter information mentioned in the embodiments below; the first parameter information refers to the related parameter information of the simulated light source itself, and the target parameter information refers to the related parameter information presented after the simulated light source irradiates the object surface. The first light distribution information can be the first light space distribution information mentioned in the embodiments below, and the second light distribution information can be the second light space distribution information mentioned in the embodiments below.

[0034] In this way, considering that the initial image (such as the first preview image) already contains the light-emitting effect caused by the ambient light and the self-luminous light, the four types of light conditions, i.e., the ambient light, the self-luminous light, the high-light reflection, and the diffuse reflection, are distinguished, and only the light space distribution information corresponding to the high-light reflection and the light space distribution information corresponding to the diffuse reflection are generated, so that the pixel value of each pixel point in the target light space distribution information is only related to the high-light reflection light intensity and the diffuse reflection light intensity, and is not related to the light intensity caused by the ambient light and the self-luminous light, thereby avoiding the problem that the light of the target lighting image is too saturated due to the repeated superposition of the light-emitting effect caused by the ambient light and the self-luminous light.

[0035] According to the first aspect or any one of the implementations of the first aspect, in the method, the performing high-light reflection light calculation according to the target parameter information to obtain the first light distribution information can include: performing high-light reflection light calculation according to the target parameter information to obtain first initial distribution information; and performing smoothing processing on the first initial distribution information to obtain the first light distribution information.

[0036] Correspondingly, the diffuse reflection light illumination is calculated according to the target parameter information to obtain second light distribution information, including: the diffuse reflection light illumination is calculated according to the target parameter information to obtain second initial distribution information; the second initial distribution information is smoothed to obtain the second light distribution information.

[0037] Exemplarily, the first initial distribution information can be the initial light space distribution information corresponding to the highlight reflection mentioned in the embodiments below, and the second initial distribution information can be the initial light space distribution information corresponding to the diffuse reflection mentioned in the embodiments below. The first initial distribution information is used to represent the highlight reflection light distribution of each unit surface in the three-dimensional model of the target object, and the second initial distribution information is used to represent the diffuse reflection light distribution of each unit surface in the three-dimensional model of the target object.

[0038] In this way, considering that the three-dimensional model is composed of multiple triangular facets, due to the discreteness of the triangular facets, the light effect displayed by the initially formed initial light space distribution information is not smooth and uniform, but presents a block and spot shape. Therefore, in order to reduce the transition difference of the light intensity between adjacent triangular facets, the light effect displayed by the light space distribution information is ensured to be smoother.

[0039] According to the first aspect, or any one of the implementations of the first aspect, in the method, the target light distribution information includes first light distribution information and second light distribution information; and the image fusion processing is performed according to the first preview image and the target light distribution information to obtain a second preview image, which can include: determining a first pixel value of a target pixel point according to the first light distribution information; and determining a second pixel value of the target pixel point according to the second light distribution information; the target pixel point is any pixel point on the three-dimensional model of the target object; performing weighted summation processing on an original pixel value, the first pixel value and the second pixel value of the target pixel point to obtain a fused pixel value; wherein the original pixel value is used to represent the pixel color of the first preview image; and the second preview image is generated according to the original pixel value of each non-target pixel point in the first preview image and the fused pixel value of each target pixel point; the non-target pixel point is a pixel point in the first preview image that is not processed by the light.

[0040] Exemplarily, each first pixel value is used to represent the highlight reflection light color of any pixel point in a triangular facet, and each second pixel value is used to represent the diffuse reflection light color of any pixel point in a triangular facet.

[0041] Thus, since the difference of the reflection light intensity of each pixel point in a certain triangle patch of the three-dimensional model of the target object is relatively small, the reflection light color of each pixel point in the certain triangle patch can be regarded as the same, and further, the first pixel value or the second pixel value of each pixel point in the certain triangle patch is regarded as the same, thereby improving the determination efficiency of the fusion pixel value of each pixel point in the three-dimensional model.

[0042] According to the first aspect, or any one of the implementations of the above first aspect, in the method, the target object is a target face, and the first preview image further includes a target human body corresponding to the target face; after the simulating light processing is performed on the three-dimensional model according to the first parameter information to obtain the target light distribution information corresponding to the three-dimensional model, the method further includes: performing human body light distribution prediction according to the target light distribution information to obtain human body light distribution information of the target human body.

[0043] Correspondingly, the image fusion processing is performed on the first preview image and the target light distribution information to obtain the second preview image, which can include: performing portrait synthesis processing according to the target light distribution information and the human body light distribution information to obtain overall light distribution information; and performing image fusion processing on the first preview image and the overall light distribution information to obtain the second preview image.

[0044] Exemplarily, the human body light distribution information can include the third light plane distribution information and the fourth light plane distribution information of the target human body mentioned in the embodiments below; and the overall light distribution information includes the first overall light distribution information and the second overall light distribution information mentioned in the embodiments below.

[0045] Thus, for the case that the initial image includes a target portrait, the target face and the target human body are segmented into two image regions, the face light distribution information (i.e., the target light distribution information) of the target face is determined first, then the human body light distribution information of the target human body is predicted based on the target light distribution information, the overall light distribution information is obtained based on the face light distribution information and the human body light distribution information, the fusion pixel value of each pixel point in the target portrait is determined based on the first preview image and the overall light distribution information, and then the second preview image (i.e., the target light image) is obtained, thereby ensuring the coordination of the light effect between the target face and the target human body in the second preview image.

[0046] In the method according to the first aspect, or any possible implementation mode of the first aspect, the target object is a target face, and the first preview image further includes a background shooting object having a positional relationship with the target face. After the simulating light processing is performed on the three-dimensional model according to the first parameter information to obtain the target light distribution information corresponding to the three-dimensional model, the method further includes: performing background light distribution prediction according to the target light distribution information and the positional relationship to obtain background light distribution information of the background shooting object.

[0047] Correspondingly, the image fusion processing is performed according to the first preview image and the target light distribution information to obtain the second preview image, which can include: performing synthesis processing according to the target light distribution information and the background light distribution information to obtain panoramic light distribution information; and performing image fusion processing on the first preview image and the panoramic light distribution information to obtain the second preview image.

[0048] Exemplarily, the background light distribution information can include the fifth light plane distribution information and the sixth light plane distribution information of the background shooting object mentioned in the embodiments below; and the panoramic light distribution information includes the first panoramic light distribution information and the second panoramic light distribution information mentioned in the embodiments below.

[0049] In this way, for the case that the initial image includes the target face and the shooting background, the target face and the background shooting object are segmented into two image regions, the face light distribution information (i.e., the target light distribution information) of the target face is determined first, then the background light distribution information of the background shooting object is predicted based on the target light distribution information, the panoramic light distribution information is obtained based on the face light distribution information and the background light distribution information, and the fusion pixel value of each pixel point in the face and the background is determined based on the first preview image and the panoramic light distribution information, so as to obtain the second preview image (i.e., the target light image), thereby ensuring the coordination of the light effect between the target face and the shooting background in the second preview image.

[0050] The first preview image, the third preview image, the first photographed image, the third photographed image, and the first image can be initial images mentioned in the embodiments below. If the initial image is the first preview image, the target lighting image corresponding to the initial image is the second preview image described above. If the initial image is the first photographed image, the target lighting image corresponding to the initial image is the second photographed image described above. If the initial image is the first image, the target lighting image corresponding to the initial image is the second image described above. If the initial image is the third preview image, the target lighting image corresponding to the initial image is the fourth preview image described above. If the initial image is the third photographed image, the target lighting image corresponding to the initial image is the fourth photographed image described above. In addition, the process of lighting processing of any one of the third preview image, the first photographed image, the third photographed image, and the first image can refer to the process of lighting processing of the first preview image, which will not be described here.

[0051] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory and, when the computer programs are executed by the one or more processors, the electronic device performs the image processing method of the first aspect and any one of the implementations of the first aspect.

[0052] The second aspect and any one of the implementations of the second aspect correspond to the first aspect and any one of the implementations of the first aspect, respectively. The technical effects corresponding to the second aspect and any one of the implementations of the second aspect can refer to the technical effects corresponding to the first aspect and any one of the implementations of the first aspect, which will not be described here.

[0053] For example, the electronic device can be a terminal device or a chip in the terminal device. The electronic device can include an input unit and a processing unit. When the electronic device is a terminal device, the processing unit can be a processor, and the input unit can be a communication interface. The terminal device can further include a memory for storing computer program code, and when the processor executes the computer program code stored in the memory, the terminal device performs any one of the image processing methods in the first aspect.

[0054] When the electronic device is a chip in a terminal device, the processing unit can be a processing unit inside the chip, and the input unit can be an output interface, a pin, or a circuit, etc. The chip can further include a memory, which can be a memory (e.g., a register, a cache, etc.) inside the chip, or a memory (e.g., a read-only memory, a random access memory, etc.) outside the chip. The memory is configured to store computer program code, and when the processor executes the computer program code stored in the memory, the chip performs the image processing method in any one of the first aspect.

[0055] In a third aspect, an embodiment of the present application provides a computer readable storage medium. The computer readable storage medium includes a computer program, and when the computer program is run on an electronic device, the electronic device executes the image processing method in the first aspect and any one of the first aspect.

[0056] The third aspect and any one of the implementation manners of the third aspect correspond to the first aspect and any one of the implementation manners of the first aspect, respectively. For details, refer to the technical effects of the first aspect and any one of the implementation manners of the first aspect, which will not be described here.

[0057] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is run, the computer executes the image processing method in the first aspect or any one of the first aspect.

[0058] The fourth aspect and any one of the implementation manners of the fourth aspect correspond to the first aspect and any one of the implementation manners of the first aspect, respectively. For details, refer to the technical effects of the first aspect and any one of the implementation manners of the first aspect, which will not be described here.

[0059] In a fifth aspect, the present application provides a chip, including a processing circuit, a receiving pin, and a sending pin. The receiving pin and the sending pin communicate with each other through an internal connection path, and the processing circuit executes the image processing method in the first aspect or any one of the first aspect to control the receiving pin to receive a signal and the sending pin to send a signal.

[0060] The fifth aspect and any one of the implementation manners of the fifth aspect correspond to the first aspect and any one of the implementation manners of the first aspect, respectively. For details, refer to the technical effects of the first aspect and any one of the implementation manners of the first aspect, which will not be described here.

[0061] It should be understood that the description of technical features, technical solutions, advantages or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and advantages described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of the specific embodiments. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 Processing flow diagram of an image processing method shown for example;

[0063] Figure 2 Processing flow diagram provided by an embodiment of the application shown for example;

[0064] Figure 3 Hardware structure diagram of an electronic device shown for example;

[0065] Figure 4 Software architecture diagram of an electronic device shown for example;

[0066] Figure 5 Application scenario diagram of an electronic device provided by an embodiment of the application;

[0067] Figure 6 Application scenario diagram of an electronic device provided by an embodiment of the application;

[0068] Figure 7 Application scenario diagram of an electronic device provided by an embodiment of the application;

[0069] Figure 8 Application scenario diagram of an electronic device provided by an embodiment of the application;

[0070] Figure 9 Application scenario diagram of an electronic device provided by an embodiment of the application;

[0071] Figure 10 Module interaction diagram of an electronic device provided by an embodiment of the application;

[0072] Figure 11 Construction process diagram of a face three-dimensional model provided by an embodiment of the application;

[0073] Figure 12 A determination process schematic diagram of a target lighting parameter of a certain unit plane provided by an embodiment of the present application;

[0074] Figure 13a A smoothing process schematic diagram of illumination space distribution information provided by an embodiment of the present application;

[0075] Figure 13b A smoothing process schematic diagram of illumination space distribution information provided by an embodiment of the present application;

[0076] Figure 14a A first frame schematic diagram of an electronic device provided by an embodiment of the present application;

[0077] Figure 14b A second frame schematic diagram of an electronic device provided by an embodiment of the present application;

[0078] Figure 14c A third frame schematic diagram of an electronic device provided by an embodiment of the present application;

[0079] Figure 14d A fourth frame schematic diagram of an electronic device provided by an embodiment of the present application;

[0080] Figure 15 A processing flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0081] Figure 16 A processing flow schematic diagram of an image processing method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0083] The term “and / or” in the present document is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone.

[0084] The terms “first” and “second” and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0085] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0086] In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless otherwise specified. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0087] For ease of understanding, some technical terms, related terms and concepts involved in some embodiments provided by the present application are explained as follows.

[0088] Face key points: refers to key feature points extracted from a face image region, which can include feature points of prominent parts such as eyes, nose, mouth, eyebrows, and facial contours.

[0089] Key point spatial distribution information of the target face: used to represent the position distribution of the face key points in the three-dimensional space, which can include three-dimensional coordinate information of the face key points.

[0090] Face solid model: traversing each face key point in the key point spatial distribution information, establishing the connection relationship between the face key points, and obtaining the face model in the three-dimensional space. The outer surface of the face solid model can include multiple face unit surfaces, and the face unit surface can be a triangular surface containing three face key points, i.e., each triangular surface can be regarded as a unit surface. The face unit surface can also be a quadrilateral containing four face key points, i.e., a certain quadrilateral can be regarded as a face unit surface.

[0091] Unit surface normal vector: refers to the normal vector of a certain face unit surface in the face solid model, and the direction of the normal vector is perpendicular to the triangular surface.

[0092] Specular reflection: when light shines on the surface of an object, the light will be concentratedly reflected to a specific direction. This phenomenon is called specular reflection.

[0093] Diffuse reflection: when light shines on the surface of an object, the light will be randomly scattered in all directions. This phenomenon is called diffuse reflection.

[0094] Specular reflection light intensity: refers to the light intensity in the direction of the specular reflection of a certain face unit surface in the face solid model.

[0095] Diffuse reflection light intensity: refers to the light intensity generated by the diffuse reflection of a unit face of a face in a face stereoscopic model. In addition, considering that the reflection direction of the diffuse reflection generated by each unit face of a face is relatively dispersed, the light intensity of the diffuse reflection generated by adjacent unit faces of a face may be projected onto a unit face of a face, therefore, the diffuse reflection light intensity of a unit face of a face can be jointly determined based on the diffuse reflection light intensity generated by the unit face itself and the diffuse reflection light intensity generated by adjacent unit faces.

[0096] Pixel value of a pixel in an image: used to represent the RGB value of a pixel. Wherein, various colors are obtained through the changes of red (Red), green (Green), and blue (Blue) three color channels and the superposition between them, and RGB represents the three channels of red, green, and blue, and the RGB value refers to the three channels of a pixel point, and is represented by an integer. Usually, each RGB value has 256, from 0, 1, 2... to 255. For example, the RGB value of a pixel point can be (122, 255, 0).

[0097] Next, the specific implementation process of some embodiments provided by the present application is described in detail.

[0098] At present, considering that photographing in a relatively dark environment light may result in an undesirable image presentation effect, in order to ensure the display effect of the obtained image, the image needs to be lighted. Referring to Figure 1 , Figure 1 is a schematic diagram of an application scenario of an image processing method. As shown in Figure 1 , an initial image to be processed is obtained; and image lighting parameter information is determined, wherein the image lighting parameter information can be default lighting parameter information of a third-party application (such as a photo editing application), or lighting parameter information set by a user through a third-party application (such as a photo editing application); based on the light addition rule corresponding to the image lighting parameter information, the to-be-added light intensity of a target pixel in the initial image is determined; based on the to-be-added light intensity, the light intensity of the target pixel in the initial image is updated to obtain a target lighted image. As can be seen, in the process of the image lighting processing, only the to-be-added light intensity of a certain image region is roughly determined based on the light addition rule, and there is a problem of low accuracy of the determination of the to-be-added light intensity, which cannot ensure the authenticity of the lighting effect of the target lighted image displayed to the user.

[0099] To address the aforementioned problems, this application provides an image processing method applied to electronic devices (such as mobile phones). This method can be used to perform lighting processing on images taken in low-light environments to obtain images with a certain lighting effect; it can also perform lighting processing on sample images to obtain sample images with different lighting effects; or it can perform lighting processing on images that require adjustment of their lighting effect to obtain images with the desired lighting effect. See also Figure 2 , Figure 2 This is a schematic diagram of the processing flow provided in an embodiment of this application. Figure 2 As shown, an initial image to be processed is acquired, which includes at least one target face. Using a pre-trained neural network model, keypoint extraction is performed on the initial image to obtain the spatial distribution information of keypoints on the target face. Then, using a neighborhood growing method, face reconstruction is performed based on the spatial distribution information of the keypoints to obtain a stereoscopic model of the target face. Next, the light source parameter information of the simulated lighting source is acquired. Using a preset lighting model, lighting calculations are performed based on the light source parameter information and the stereoscopic face model to obtain the first spatial distribution information of illumination corresponding to specular reflection and the second spatial distribution information of illumination corresponding to diffuse reflection. Finally, image fusion processing is performed based on the initial image and the first and second spatial distribution information of illumination to obtain the target illuminated image.

[0100] In the image processing method provided in this application embodiment, facial key points are first extracted from the initial image. Then, a three-dimensional facial model is constructed based on the facial key points in three-dimensional space. Next, based on the three-dimensional facial model and the light source parameter information of the simulated lighting source, target illumination spatial distribution information (such as illumination spatial distribution information corresponding to specular reflection and illumination spatial distribution information corresponding to diffuse reflection) is generated. Finally, image fusion is performed between the target illumination spatial distribution information and the initial image to obtain the target illuminated image. Based on this, the technical effects produced by the embodiments provided in this application include at least the following:

[0101] On the one hand, by using a preset lighting model, the 3D face model constructed based on facial key points is simulated with lighting, and the intensity of reflected light is calculated. Then, the spatial distribution information of target lighting is generated. This can simulate the lighting direction and intensity corresponding to each unit surface in the 3D face model, making the lighting effect added to the initial image closer to the real lighting effect, thereby ensuring the authenticity of the target lighting image shown to the user.

[0102] In another aspect, for the construction process of the face three-dimensional model, the position information of the face key points obtained by using the neural network model not only includes the plane coordinate information of the face key points, but also includes the depth information of the face key points, and then the key point space distribution information of the target face is obtained. In this way, the connection relationship between the face key points can be directly established based on the key point space distribution information, the face three-dimensional model is obtained, and the construction efficiency of the face three-dimensional model is improved.

[0103] In another aspect, for the generation process of the light space distribution information, the face three-dimensional model is obtained by using the domain growing method to establish the connection relationship between the face key points, so that the face three-dimensional model can include a plurality of unit faces. Each unit face is taken as a minimum calculation unit to calculate the reflected light intensity corresponding to each unit face in the face three-dimensional model, and then the pixel value of each pixel point on the unit face is determined, and then the target light space distribution information is generated based on the pixel value. In this way, not only the calculation accuracy of the reflected light intensity can be ensured, but also the generation efficiency of the light space distribution information can be improved.

[0104] In addition, for the image fusion process, considering that the initial image already contains the light-emitting effect caused by the ambient light and the self-luminous light, the four types of light conditions, i.e., the ambient light, the self-luminous light, the highlight reflection and the diffuse reflection, are distinguished, and only the light space distribution information corresponding to the highlight reflection and the light space distribution information corresponding to the diffuse reflection are generated. The pixel value of each pixel point in the target light space distribution information is only related to the highlight reflection light intensity and the diffuse reflection light intensity, and is not related to the light intensity generated by the ambient light and the light intensity generated by the self-luminous light. Therefore, the problem that the light of the target light image is too saturated due to the repeated superposition of the light-emitting effect caused by the ambient light and the self-luminous light is avoided.

[0105] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0106] The technical solutions provided in the present application can be applied in an electronic device with a shooting function. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a handheld computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a netbook, and an electronic device such as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, and / or a smart city device, and the like. The specific type of the electronic device is not specially limited in the embodiments of the present application.

[0107] First, the hardware structure of the electronic device is described in detail.

[0108] For example, the electronic device is a mobile phone, Figure 3 The hardware structure of the electronic device is shown. It should be understood that, Figure 3 The electronic device 100 shown is only an example of an electronic device, and the electronic device 100 can have more or fewer components than shown in the figure, can combine two or more components, or can have a different component configuration. Figure 3 The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0109] As Figure 3As shown, the electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0110] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0111] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0112] The memory in the processor 110 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0113] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0114] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or combinations of multiple interface connection modes in the above embodiments.

[0115] The wireless communication function of the electronic device 100 can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0116] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0117] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0118] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0119] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0120] In some embodiments, the display screen 194 can be configured to display a page (e.g., a photographing interface) required by the electronic device 100, and display an image captured by any one or more of the cameras 193 in the page.

[0121] The electronic device 100 can implement the photographing function through an ISP, the cameras 193, a video codec, a GPU, the display screen 194, and an application processor, or the like.

[0122] The ISP is configured to process data fed back by the cameras 193. For example, when taking a photograph, a shutter is opened, light is transmitted to a camera photosensitive element through a lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing, and converts the electrical signal into an image visible to the naked eye. The ISP can also optimize algorithms for noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of a photographing scene. In some embodiments, the ISP can be disposed in the cameras 193.

[0123] The camera 193 is used to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to an ISP to convert into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in formats such as RGB, YUV, etc. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0124] In embodiments of the present application, the type of camera 193 can be distinguished according to the hardware configuration and the physical location. For example, a camera disposed on the side of the display screen 194 of the electronic device can be referred to as a front-facing camera, and a camera disposed on the side of the back cover of the electronic device can be referred to as a rear-facing camera. For another example, a camera with a short focal length and a large field of view can be referred to as a wide-angle camera, and a camera with a long focal length and a small field of view can be referred to as a normal camera. The length of the focal length and the size of the field of view are relative concepts and are not limited by specific parameters, so the wide-angle camera and the normal camera are also a relative concept, and can be distinguished according to physical parameters such as focal length and field of view.

[0125] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is in frequency selection, the digital signal processor is used to perform Fourier transform on frequency energy, etc.

[0126] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0127] The NPU is a neural-network (NN) computing processor, which learns from the structure of biological neural networks, such as the transmission mode between human brain neurons, to quickly process input information and can also continuously self-learn. Through the NPU, the electronic device 100 can realize intelligent cognition applications such as image recognition, face recognition, voice recognition, text understanding, etc.

[0128] Next, the software architecture of the electronic device 100 is described in detail.

[0129] The software system of the electronic device 100 can employ a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. Embodiments of the present application take an Android system with a layered architecture as an example to illustrate the software architecture of the electronic device 100. Figure 4 A software architecture diagram of the electronic device 100 of embodiments of the present application is shown. It can be understood that, Figure 4 The layers in the software architecture shown and the components contained in each layer are only an example and do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer layers than shown, and each layer can include more or fewer components, or combine certain components, or split certain components, or different component arrangements, which are not limited by the present application.

[0130] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer (HAL layer), and the kernel layer.

[0131] The application layer can include a series of application packages. As Figure 4 shown, the application package can include camera, gallery, WLAN, Bluetooth, and other applications (APPs). The application package can also include calling, calendar, map, navigation, music, video, short message, and other applications.

[0132] In embodiments of the present application, a certain application with a photographing function (such as a camera application) responds to a user's trigger operation, calls the camera HAL of the hardware abstraction layer through the camera service of the application framework layer, controls the camera driver of the kernel layer, and thus controls the camera to take the photos or videos required by the user.

[0133] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0134] As Figure 4 shown, the application framework layer can include a window manager, a content provider, a resource manager, a view system, a notification manager, and the like. The application framework layer can also include a camera service and an image processing service.

[0135] The window manager is used to manage windows programs. The window manager can acquire the display screen size, determine whether there is a status bar, lock the screen, intercept the screen, etc.

[0136] The content provider is used to store and acquire data, and make the data accessible to the application program. The data can include video, image, audio, dialed and received phone, browsing history and bookmark, phone book, etc.

[0137] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build an application program. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0138] The resource manager provides various resources for the application program, such as localized strings, icons, pictures, layout files, video files, etc.

[0139] The notification manager enables the application program to display notification information in the status bar, which can be used to convey a type of message that can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of downloading, message reminders, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application program running in the background, and can also be a notification in the form of a dialog window appearing on the screen. For example, a text message is prompted in the status bar, a prompt sound is emitted, the electronic device vibrates, the indicator light flashes, etc.

[0140] In the embodiment of the application, the camera service is used to control the camera through the camera HAL of the hardware abstraction layer and the camera driver of the kernel layer, so as to obtain an initial image collected by the camera in real time. In the case that the application processor performs image lighting processing, the image processing service is used to perform lighting processing on the initial image to obtain a target lighting image.

[0141] Illustratively, the image processing service can specifically include a key point extraction module, a stereo model construction module, a light distribution determination module, and an image fusion module. The key point extraction module is used to perform key point extraction processing on a target shooting object (such as a target face) in the initial image to obtain key point spatial distribution information of the target shooting object. The stereo model construction module is used to perform three-dimensional reconstruction based on the key point spatial distribution information to obtain a stereo model (such as a face stereo model) of the target shooting object. The light distribution determination module is used to perform simulated lighting processing on the stereo model of the target shooting object to obtain light spatial distribution information of the target shooting object. The image fusion module is used to perform image fusion processing based on the initial image and the light spatial distribution information of the target shooting object to obtain the target lighting image.

[0142] The Android Runtime includes a core library and a virtual machine. The Android Runtime is responsible for scheduling and managing the Android system.

[0143] The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0144] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection and the like.

[0145] The system library can include a plurality of functional modules. For example: a surface manager, media libraries, a three-dimensional graphics processing library (for example: OpenGL ES), a 2D graphics engine (for example: SGL) and the like.

[0146] The surface manager is used to manage the display subsystem, and provides a plurality of applications with the fusion of 2D and 3D layers.

[0147] The media library supports a plurality of commonly used audio, video format playback and recording, and static image files and the like. The media library can support a plurality of audio and video coding formats, for example: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG and the like.

[0148] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing and the like. The 2D graphics engine is a drawing engine for 2D drawing.

[0149] The hardware abstraction layer (or HAL layer) is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of the specific platform, provides a virtual hardware platform for the operating system, and makes it hardware-independent, which can be ported on multiple platforms. The HAL layer provides a standard interface to display device hardware functions to the higher-level Java API framework (i.e., the framework layer). The HAL layer includes multiple library modules, each of which implements an interface for a specific type of hardware component, such as: audio HAL audio module (or audio HAL, audio hardware abstraction module), Bluetooth HAL Bluetooth module (or Bluetooth HAL, Bluetooth hardware abstraction module), camera HAL camera module (also referred to as camera HAL, camera hardware abstraction module), sensors HAL sensor module (or sensor HAL, Isensor service, sensor service).

[0150] In the embodiments of the present application, the camera HAL is mainly used to play a role of connecting the upper and lower layers, and can provide its own methods (or functions or APIs) to the camera service through the HIDL interface of the HAL layer, so that the camera service can communicate with the underlying driver (i.e., the instructions of the camera service can be transmitted to the camera module, so that the camera module works according to the instructions). In this way, the application program can control the camera driver through the camera service calling the camera HAL, thereby achieving the purpose of controlling the camera to take pictures. In the case of performing image lighting processing by the digital signal processor, the camera HAL is also used to receive the initial image from the application framework layer and send the initial image to the digital signal processor to trigger the digital signal processor to perform lighting processing on the initial image to obtain the target lighting image.

[0151] Correspondingly, the digital signal processor can include a key point extraction module, a stereo model construction module, a light distribution determination module, and an image fusion module. The key point extraction module is configured to perform key point extraction processing on a target photographed object (such as a target face) in the initial image to obtain key point spatial distribution information of the target photographed object. The stereo model construction module is configured to perform three-dimensional reconstruction based on the key point spatial distribution information to obtain a stereo model (such as a face stereo model) of the target photographed object. The light distribution determination module is configured to perform simulated lighting processing on the stereo model of the target photographed object to obtain light spatial distribution information of the target photographed object. The image fusion module is configured to perform image fusion processing based on the initial image and the light spatial distribution information of the target photographed object to obtain the target lighting image.

[0152] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, camera drivers, Bluetooth drivers, sensor drivers, etc. The hardware at least includes processors, display screens, cameras, Bluetooth modules, sensors, etc.

[0153] The display driver can drive the display screen in the electronic device to display.

[0154] The technical solutions provided in the embodiments of the present application can be implemented in the electronic device with the hardware architecture or software architecture described above.

[0155] It can be understood that, in order to implement the image processing method in the embodiments of the present application, the electronic device contains the hardware and / or software modules corresponding to the execution of each function. The algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0156] For ease of understanding, before the image processing method provided by the embodiments of the present application is described in detail, the time node of triggering the execution of the initial image lighting processing process related to the embodiments of the present application is described, Figures 5 to 9An application scenario of the electronic device is shown in the figure. In this example, the electronic device is a mobile phone, and the target application is a camera application. In one case, during the photograph preview process, if it is detected that the user clicks the light control, the photograph image collected by the camera is not stored, but the target light image corresponding to the photograph image is directly stored in a specified location, and the target light image is viewed through the gallery application. For example, before the user clicks the photograph control, if it is detected that the user clicks the light control, the preview image to be displayed is first lighted to obtain the target light image, and the target light image is displayed in the preview area of the photograph preview interface of the camera application, so that the user can determine whether the light effect of the target light image meets the expectation. In the case where the user confirms that the image light effect meets the expectation, the photograph control can be clicked. If it is detected that the user clicks the photograph control, the photograph image is collected by the camera, the photograph image is lighted to obtain the corresponding target light image, and the target light image is stored. That is, for the photographing scene, if it is detected that the user clicks the light control, the preview image needs to be lighted before the user clicks the photograph control, and the target light image corresponding to the preview image is displayed in the preview area. After the user clicks the photograph control, the photograph image needs to be lighted, and the target light image corresponding to the photograph image is stored. In another case, a historical photograph image can be selected from the gallery application as an image to be lighted, the light processing of the historical photograph image is triggered to obtain the target light image, and the target light image is stored in a specified location and viewed through the gallery application.

[0157] In some example embodiments, as shown in Figure 5 , for the initial image to be lighted is a real-time collected preview image, referring to (a) in Figure 5 , the mobile phone desktop is displayed, and the user clicks the application icon 501 of the camera application. The mobile phone responds to the user operation and displays the photograph preview interface as shown in (b) in Figure 5 . The preview area 502 in the photograph preview interface displays the image data of the photographed object collected by the camera of the electronic device at present. At this time, the user can click the first light control 503 in the photograph preview interface. The mobile phone responds to the click operation, acquires the next frame of preview image to be displayed, and acquires the light source parameter information of the simulated light source. The light source parameter information can be automatically determined by the system. The preview image to be displayed is lighted based on the light source parameter information to obtain the first light image (i.e., the target light image corresponding to the preview image). Wherein, the preview image to be displayed is not Figure 5The preview image shown in (b) is displayed in preview area 502, but the next frame preview image to be displayed is the image that will be displayed after a user click operation on the first lighting control 503 is detected. Next, the image will be displayed as shown in (b). Figure 5 As shown in (c) of the image preview interface, the preview area 502 displays the first lighting image. If the lighting effect of the first lighting image meets the user's expectations, the user can click the camera control 504. The phone responds to the click operation, captures the image through the camera, performs lighting processing on the captured image to obtain the second lighting image (i.e., the target lighting image corresponding to the captured image), and stores the second lighting image. The user can quickly view the second lighting image from the gallery by clicking the photo viewing control in the lower left corner of the image preview interface.

[0158] In addition, such as Figure 6 As shown, a lighting parameter setting interface can be provided to users in photography scenarios. (See [link]) Figure 6 In step (a), the phone's home screen is displayed, and the user clicks the camera app icon 501; the phone responds to this user action and displays as shown below. Figure 6 As shown in (b) of the image preview interface, the preview area 502 of the image preview interface displays the image data of the subject currently captured by the camera of the electronic device. At this time, the user can click the first lighting control 503 in the image preview interface; the mobile phone responds to the user's operation and displays as shown in the image preview interface. Figure 6 In the lighting parameter setting interface shown in (c), the user can customize the light source parameter information of the simulated lighting source by triggering the first parameter setting control 505. Then, the user clicks the first confirmation control 506, indicating that the lighting parameter setting is complete. The phone responds to this click by acquiring the next frame of the preview image to be displayed, as well as the light source parameter information of the simulated lighting source. Based on this light source parameter information, lighting processing is performed on the preview image to be displayed to obtain the first lit image. Then, the image is displayed as shown... Figure 6 The photo preview interface shown in (d) displays a first lighting image in the preview area 502. If the user confirms that the lighting effect of the first lighting image meets expectations, the user can click the photo control 504. The mobile phone responds to the click operation, captures the image through the camera, and stores the second lighting image corresponding to the captured image.

[0159] Furthermore, such as Figure 7 As shown, for a scenario where the user triggers adjustments to the image lighting effect during the photo preview process, for example, taking the image relighting triggered by the first lighting control 503 as an example, see [link to example]. Figure 7The photo preview interface shown in (a) displays a first illuminated image in preview area 502. If the user confirms that the lighting effect of the first illuminated image does not meet expectations, the user can click the first illuminated control 503. The phone responds to this click operation by displaying... Figure 7 In the lighting parameter setting interface shown in (b), the user can adjust the light source parameters of the simulated lighting source by triggering the first parameter setting control 505. Then, the user clicks the first confirmation control 506, indicating that the lighting parameters have been reset. The phone responds to this click by acquiring the next frame of the preview image to be displayed, as well as the adjusted light source parameters. Based on these adjusted light source parameters, the preview image to be displayed is processed with lighting to obtain the adjusted lighting image. Then, the image is displayed as shown... Figure 7 The photo preview interface shown in (c) displays the adjusted lighting image in the preview area 502. If the user confirms that the lighting effect of the adjusted lighting image meets expectations, the user can click the photo control 504. The mobile phone responds to the click operation, captures the image through the camera, performs lighting processing on the captured image based on the adjusted light source parameter information, and stores the target lighting image corresponding to the captured image.

[0160] Furthermore, such as Figure 8 As shown, the newly added lighting effect input control can be used to adjust the image lighting effect. For example, taking the image relighting triggered by the human-computer interaction control 507 as an example, see [link to example]. Figure 8 The photo preview interface shown in (a) displays a first illuminated image in preview area 502. If the user confirms that the lighting effect of the first illuminated image does not meet expectations, the user can click the human-computer interaction control 507. The phone responds to this click operation by displaying the following... Figure 8 The human-computer interaction interface 508 shown in (b) allows the user to input a description of the desired lighting effect by triggering the information input control 509. Then, the system automatically performs parameter matching based on the desired lighting effect description to obtain adjusted light source parameter information. Next, based on this adjusted light source parameter information, the next frame of the preview image to be displayed is lit to obtain an adjusted lighting image. Then, the image is displayed as shown in (b). Figure 8 The photo preview interface shown in (c) displays the adjusted lighting image in the preview area 502. If the user confirms that the lighting effect of the adjusted lighting image meets expectations, the user can click the photo control 504. The mobile phone responds to the click operation, captures the image through the camera, performs lighting processing on the captured image based on the adjusted light source parameter information, and stores the target lighting image corresponding to the captured image.

[0161] In the embodiments of the present application, the man-machine interaction control 507 different from the first light control 503 is added, and after detecting the triggering operation of the man-machine interaction control 507 by the user, the man-machine interaction interface 508 is displayed; wherein the man-machine interaction interface 508 can be a display interface of a voice assistant application. Based on this, after detecting the triggering operation of the man-machine interaction control 507 by the user, the voice assistant application is started, the man-machine interaction interface is displayed, and the expected light effect description information input by the user is listened to. Therefore, the adjusted light source parameter information can be obtained by matching the parameter values based on the expected light effect description information input by the user. Therefore, for the user who does not understand the corresponding relationship between the light parameter value and the light effect, the user only needs to describe the expected light effect, and does not need to pay attention to the specific value of each light parameter.

[0162] In addition, it can be understood that the man-machine interaction interface can also be triggered by the first light control 503, for example, in the case of displaying the lighted image in the preview area, if it is detected that the user clicks the first light control 503 again, it indicates that the user is not satisfied with the light effect of the displayed lighted image, and has the demand of triggering the image to be lighted again. At this time, the man-machine interaction interface can be directly entered, so that the user only needs to input the description information of the expected light effect, thereby improving the adjustment effect of the light source parameter information.

[0163] In some other example embodiments, as shown in Figure 9 for the initial image to be lighted for processing is a historical photographed image, referring to (a) in Figure 9 , the mobile phone desktop is displayed, the user clicks the application icon 601 of the gallery application; the mobile phone responds to the user operation, and displays the photo display interface 602 as shown in (b) in Figure 9 . After the user selects a photo as a gallery image, the user can perform a drag operation on the gallery image to the direction of the second light control 603 to indicate that the gallery image is lighted for processing; the mobile phone responds to the user operation, and displays the light parameter setting interface as shown in (c) in Figure 9 . The user sets the parameter values of the light parameters by clicking the second parameter setting control 604, and then clicks the second confirmation control 605, which indicates that the light parameter setting is completed; the mobile phone responds to the clicking operation, and obtains the gallery image selected by the user and the light source parameter information of the simulated light source; the gallery image is lighted for processing based on the light source parameter information, and a third lighted image is obtained. Next, as shown in Figure 9In the image editing interface shown in (d) in the figure, the third lighted image and the save control 606 are displayed on the image editing interface. At this time, if the user confirms that the lighted effect of the third lighted image meets the expectation, the user can click the save control 606 to store the third lighted image. The user can enter the all photo display interface to view the third lighted image. The user can also share, edit, delete, and the like, the third lighted image.

[0164] In addition, if the user confirms that the lighted effect of the third lighted image does not meet the expectation, the image can also be re-lighted by triggering the specified control. The determination process of the adjusted light source parameter information can refer to the specific implementation process described above in (d) in the figure, which will not be described herein again. Figure 7 and Figure 8 The specific implementation process described above in (d) in the figure will not be described herein again.

[0165] It can be understood that, for the determination process of the light source parameter information used for light processing of the image, after detecting the click operation of the user on the light control (such as the first light control 503 or the second light control 603), the initial image can be directly lighted based on the default light source parameter information. In this way, after the user clicks the light control, the target lighted image is directly displayed, and the light parameter setting interface is not displayed. The default light source parameter information can be the light source parameter information set by the user in advance, the light source parameter information automatically recommended by the system, or the light source parameter information used by the user last time. For example, after detecting the click operation of the user on the light control, the mobile phone automatically obtains the default values of the light source type, the light source intensity, the light source color, the light source center position, and the light source direction. Then, the image is lighted based on the default values of the parameter items to obtain the lighted image. Alternatively, after detecting the click operation of the user on the light control (such as the first light control 503 or the second light control 603), a light parameter setting interface can also be provided for the user to set the light parameters according to the actual needs. The human-computer interaction interface can also be triggered by the specified control to match the parameter values based on the expected lighted effect description information input by the user to obtain the light source parameter information of the simulated light source.

[0166] Next, the specific implementation process of the image processing method provided by the embodiment of the present application will be described in combination with a specific application scenario example. In the following, the electronic device is taken as a mobile phone, and the target application is taken as a camera application for example to explain and describe. The embodiment of the present application will not be described again for other types of electronic devices and other applications with photographing functions.

[0167] Scenario One

[0168] In the present scenario, taking an initial image containing a target face that needs to increase the lighting effect as an example, the target photographic object to be lighted can be a target face. For example, the present scenario is a scenario in which a target user uses a mobile phone to take a selfie, and the image region of the target face in the initial image is lighted to obtain a target lighted image. As shown in Figure 10 The specific implementation process of generating the target lighted image corresponding to the initial image can include the following steps:

[0169] In S701, the key point extraction module uses a pre-trained face key point detection model to perform key point extraction processing on the initial image to obtain key point spatial distribution information of the target face.

[0170] The target application can include any one of a camera application, a gallery application, and other applications with a photographing function; the target application sends an initial image to be lighted to the key point extraction module; the key point extraction module obtains the initial image; and the initial image includes an image region of a target face. In an example, in one case, the initial image can be a preview image to be displayed on a photographing preview interface of the camera application; the mobile phone detects a triggering operation of a user on an application icon of the camera application, starts the camera application, and collects a preview image through a camera. In another case, the initial image can also be a photographing image; the mobile phone detects a triggering operation of a user on a photographing control of the camera application, and collects a photographing image through the camera. In yet another case, the initial image can also be a historical photographing image from the gallery application; the mobile phone detects a triggering operation of a user on the gallery application, and determines a photo selected by the user from the gallery application interface as the initial image. After the target application obtains the initial image, the target application sends the initial image to the key point extraction module and the image fusion module.

[0171] In an example, the face key point detection model is obtained by pre-iteratively updating a neural network model based on a face image sample set. The face key point detection model not only learns to identify the planar coordinates of the face key points, but also learns to identify the depth information of the face key points, i.e., learns the spatial distribution of the key feature points of different parts of the target face. Therefore, the position information of each face key point in the key point spatial distribution information includes not only the planar coordinate information of the face key point, but also the depth information of the face key point.

[0172] After the key point extraction module generates the key point spatial distribution information of the target face, the key point spatial distribution information is sent to the stereoscopic model construction module, so that the stereoscopic model construction module constructs a face stereoscopic model of the target face according to the key point spatial distribution information.

[0173] S702, the stereoscopic model construction module uses the neighborhood growing method to perform three-dimensional reconstruction processing based on the spatial distribution information of the key points of the target face, to obtain a stereoscopic model of the target face; taking the construction of a face unit surface from three adjacent face key points as an example, the stereoscopic model of the face includes multiple triangular facets.

[0174] For example, the face key point with the maximum depth is determined as the starting growing point, and the face key point on the face boundary contour line is determined as the ending growing point. The starting growing point is taken as the first selected current growing point, the adjacent growing points of the current growing point are determined, and the connection relationship between the current growing point and the adjacent growing points is established; the adjacent growing points are taken as the next selected current growing point, the adjacent growing points of the next current growing point are determined, and the connection relationship between the next current growing point and the adjacent growing points is established; until the current growing point is the ending growing point and each face key point is selected as a current growing point; the key point spatial distribution information with the established connection relationship is determined as the stereoscopic model of the target face.

[0175] For example, as shown in Figure 11 , the specific implementation process of growing from the starting growing point to the ending growing point to obtain the stereoscopic model of the face is given, referring to Figure 11 (a), the spatial distribution information of the key points of the target face is shown; referring to Figure 11 (b), a starting growing point A0 (i.e. the key point at the highest part of the nose bridge) and multiple ending growing points B (i.e. the face boundary key points, that is, the light gray key points on the outermost edge enclosed by the dashed line) are determined from the multiple face key points in the key point spatial distribution information. Referring to Figure 11 (c), the starting growing point A0 is taken as the first selected current growing point, the adjacent growing points (such as A1 to A6) of the current growing point are determined, and the connection relationship between the current growing point and the adjacent growing points is established. Referring to Figure 11 (d), the adjacent growing points are taken as the next selected current growing point, the adjacent growing points of the next current growing point are determined, and the connection relationship between the next current growing point and the adjacent growing points is established. Referring to Figure 11 (e), until the current growing point is the ending growing point B and each face key point is selected as a current growing point; the key point spatial distribution information with the established connection relationship is determined as the stereoscopic model of the target face.

[0176] The stereoscopic model construction module constructs a face stereoscopic model of the target face, and sends the face stereoscopic model to the light distribution determination module, so that the light distribution determination module performs simulated lighting processing on the face stereoscopic model according to the light source parameter information of the simulated light source, and obtains light space distribution information of different reflection types.

[0177] In S703, the light distribution determination module determines target light parameter information of each triangular patch in the face stereoscopic model based on the light source parameter information of the simulated lighting light source.

[0178] The target application sends the light source parameter information to the light distribution determination module after determining the light source parameter information of the simulated lighting light source. The light source parameter information includes at least one of the light source type, the light source luminous intensity, the light source color, the light source direction, and the light source center position information.

[0179] In some example embodiments, the light source parameter information of the simulated lighting light source can be default light source parameter information. In other example embodiments, the light source parameter information can also be determined based on user parameter setting confirmation information after the mobile phone detects a user trigger operation on the lighting control and enters the lighting parameter setting interface. Specifically, the light source parameter information can be determined based on user confirmation operation of automatically recommended light parameter items, or can be user-defined light source parameter information. In yet other example embodiments, the light source parameter information can also be determined by matching parameter values based on user input description information of the desired lighting effect after the mobile phone detects a user trigger operation on the specified control and enters the human-computer interaction interface.

[0180] The target application sends the light source parameter information to the light distribution determination module, so that the light distribution determination module determines the light space distribution information of the target face according to the light source parameter information.

[0181] Exemplarily, each triangular patch is taken as a face unit surface, and for each face unit surface, the target light parameter information corresponding to the face unit surface is determined based on the light source parameter information of the simulated light source. The target light parameter information includes a light source incident normal vector and a unit surface normal vector. The light source incident normal vector of different triangular patches can be different, and the unit surface normal vector of different triangular patches can also be different.

[0182] Exemplarily, as shown in Figure 12 A schematic diagram of determining target light parameter information of any triangular patch by taking the triangular patch in the face stereoscopic model as the smallest calculation unit is given, as shown in Figure 12The direction of the unit surface normal vector of different triangular patches in the face three-dimensional model is different, as shown in (a) of FIG. 7. The unit surface normal vector can be determined based on the three-dimensional coordinate information of the triangular patch, and the direction of the unit surface normal vector is perpendicular to the plane where the triangular patch is located. See Figure 12 (b) of FIG. 7 shows a cross-sectional view of the face three-dimensional model, where L indicates the direction of the light source incident normal vector of a triangular patch, N indicates the direction of the unit surface normal vector of a triangular patch, V indicates the observation direction, and R indicates the highlight reflection direction of a triangular patch.

[0183] In S704, the light distribution determination module determines the highlight reflection light intensity of each triangular patch in the face three-dimensional model based on the first light intensity calculation formula and the target light parameter information corresponding to the triangular patch.

[0184] For example, the first light intensity calculation formula can be expressed as:

[0185]

[0186] where c s represents the highlight reflection light intensity of a triangular patch, k represents the light attenuation coefficient, c l represents the light source luminous intensity, m s represents the highlight reflection coefficient, represents the observation direction normal vector, represents the unit surface normal vector of a triangular patch, represents the light source incident normal vector of a triangular patch, m g represents the skin reflection coefficient of the target face.

[0187] It can be understood that the observation direction normal vector, the skin reflection coefficient of the target face, and the light attenuation coefficient, the light source luminous intensity, and the highlight reflection coefficient are all known parameters.

[0188] In S705, the light distribution determination module generates the first light space distribution information corresponding to the highlight reflection based on the highlight reflection light intensity of each triangular patch in the face three-dimensional model.

[0189] Specifically, first, for each triangular patch in the face three-dimensional model, the first pixel value of each pixel point in the triangular patch is determined based on the first mapping relationship and the highlight reflection light intensity corresponding to the triangular patch; each first pixel value is used to represent the highlight reflection light color of any pixel point in a triangular patch. Then, the first light space distribution information corresponding to the highlight reflection is generated based on the first pixel value of each pixel point in the face three-dimensional model.

[0190] Exemplarily, the first mapping relationship includes a correspondence between the highlight reflection light intensity and the highlight reflection light color. Wherein, since the difference of the highlight reflection light intensity of each pixel point in a certain triangle patch is relatively small, the highlight reflection light color of each pixel point in a certain triangle patch can be regarded as the same, and further, the first pixel value of each pixel point in a certain triangle patch is regarded as the same.

[0191] Exemplarily, the first light space distribution information is used to represent the distribution of the highlight reflection light color of each pixel point in the face three-dimensional model. In some example embodiments, considering that the face three-dimensional model is composed of a plurality of triangle patches, due to the discreteness of the triangle patches, the light effect displayed by the initially formed initial light space distribution information is not smooth and uniform distribution, but presents a block spot shape. Therefore, in order to reduce the transition difference of the highlight reflection light intensity (corresponding to the first pixel value) between adjacent triangle patches, so as to ensure that the light effect displayed by the first light space distribution information is more smooth. Based on this, as shown in the following formula, first, the initial light space distribution information corresponding to the highlight reflection is generated based on the first pixel value of each pixel point in the face three-dimensional model; and then the pixel smoothing processing is performed on the initial light space distribution information to obtain the first light space distribution information. Figure 13a

[0192] S706, the light distribution determination module determines the diffuse reflection light intensity of each triangle patch in the face three-dimensional model based on the second light intensity calculation formula and the target light parameter information corresponding to the triangle patch.

[0193] Exemplarily, the second light intensity calculation formula can be expressed as:

[0194]

[0195] Wherein, c d represents the diffuse reflection light intensity of a certain triangle patch, k represents the light attenuation coefficient, c l represents the light source luminous intensity, m d diffuse reflection coefficient, represents the unit surface normal vector of a certain triangle patch, represents the light source incident normal vector of a certain triangle patch.

[0196] It can be understood that the light attenuation coefficient, the light source luminous intensity and the diffuse reflection coefficient are all known parameters. In addition, considering that the reflection direction of the diffuse reflection generated by each triangle patch is relatively dispersed, the light intensity of the diffuse reflection generated by the adjacent triangle patches can be projected on a certain triangle patch, therefore, the diffuse reflection light intensity of a certain triangle patch can be jointly determined based on the component of the light intensity of the diffuse reflection generated by the triangle patch itself and the light intensity of the diffuse reflection generated by the adjacent triangle patches.​

[0197] S707, the light distribution determination module generates second light space distribution information corresponding to diffuse reflection based on the diffuse reflection light intensity of each triangular facet in the face stereoscopic model.

[0198] Specifically, first, for each triangular facet in the face stereoscopic model, based on the second mapping relationship and the diffuse reflection light intensity corresponding to the triangular facet, the second pixel value of each pixel point in the triangular facet is determined; each second pixel value is used to represent the diffuse reflection light color of any pixel point in a triangular facet. Then, based on the second pixel value of each pixel point in the face stereoscopic model, the second light space distribution information corresponding to diffuse reflection is generated.

[0199] Illustratively, the second mapping relationship includes the correspondence between the diffuse reflection light intensity and the diffuse reflection light color. Wherein, since the difference of the diffuse reflection light intensity of each pixel point in a certain triangular facet is relatively small, the diffuse reflection light color of each pixel point in a certain triangular facet can be considered the same, and further, the second pixel value of each pixel point in a certain triangular facet is considered the same.

[0200] Illustratively, the second light space distribution information is used to represent the distribution of the diffuse reflection light color of each pixel point in the face stereoscopic model. In some example embodiments, considering that the face stereoscopic model is composed of multiple triangular facets, due to the discreteness of the triangular facets, the light effect exhibited by the initially formed initial light space distribution information is not smooth and uniform distribution, but presents a block spot shape, therefore, in order to reduce the transition difference of the diffuse reflection light intensity (corresponding to the second pixel value) between adjacent triangular facets, so as to ensure that the light effect exhibited by the second light space distribution information is more smooth. Based on this, as shown in the figure, first, based on the second pixel value of each pixel point in the face stereoscopic model, the initial light space distribution information corresponding to diffuse reflection is generated; then the pixel smoothing processing is performed on the initial light space distribution information, to obtain the second light space distribution information. Figure 13b

[0201] Wherein, after the light distribution determination module generates the first light space distribution information and the second light space distribution information, the first light space distribution information and the second light space distribution information are sent to the image fusion module, so that the image fusion module performs image fusion processing according to the first light space distribution information and the second light space distribution information, to obtain the target lighting image.

[0202] S708, the image fusion module performs image fusion processing based on the initial image, the first light space distribution information and the second light space distribution information, to obtain the target lighting image.

[0203] ​Specifically, first pixel values of target pixel points in the initial image are determined according to the first light space distribution information, and second pixel values of the target pixel points in the initial image are determined according to the second light space distribution information; the target pixel points are any pixel points on the face three-dimensional model; then, the original pixel values, the first pixel values and the second pixel values of the target pixel points are weighted and summed to obtain fusion pixel values of the target pixel points; wherein the original pixel values are used to represent the pixel colors of the initial image; next, a target lighting image is generated according to the original pixel values of each non-target pixel point in the initial image and the fusion pixel values of each target pixel point; the non-target pixel points are pixel points in the initial image that are not processed by lighting.

[0204] For example, the image fusion process can be understood as a weighted sum calculation of pixel values of pixel points, and the fusion pixel values of each pixel point in the target lighting image are calculated based on a preset image fusion formula; wherein the image fusion formula can be expressed as:

[0205] K i,j = min(m1C i,j +m2B i,j +m3M i,j ,C max )

[0206] Wherein, K i,j represents the fusion pixel value of the pixel point (i, j), C i,j represents the original pixel value of the pixel point (i, j) in the initial image, B i,j represents the first pixel value of the pixel point (i, j), M i,j represents the second pixel value of the pixel point (i, j), C max represents the maximum pixel value of the pixel point (i, j) in the target lighting image, m1 represents the fusion weight coefficient corresponding to the initial image, m2 represents the fusion weight coefficient of the highlight reflection light color, and m3 represents the fusion weight coefficient of the diffuse reflection light color.

[0207] It can be understood that for non-target pixel points in the initial image, the first pixel value is equal to zero, and the second pixel value is equal to zero, therefore, the fusion pixel value of the non-target pixel point is equal to the original pixel value.

[0208] Next, the image fusion module sends the target lighting image to the target application. After receiving the target lighting image, the target application displays the target lighting image so that the user confirms whether the lighting effect of the target lighting image meets the expectation.

[0209] It should be noted that the specific implementation process of the image lighting processing can be implemented by a function module in the application framework layer, or can be implemented by a function module in the hardware abstraction layer, or can be implemented by a function module in the digital signal processor, and the present application does not make any limitation in this regard.

[0210] In addition, the specific implementation process of the image lighting processing can be used to perform lighting processing on the initial sample image (corresponding to the initial image) to generate a target sample image with diversified lighting effects (corresponding to the target lighting image); and then based on the sample image set, the to-be-trained model is iteratively trained to obtain the trained image recognition model, so as to ensure the recognition accuracy of the image recognition model for images with different lighting effects. Or because the on-site light direction does not meet the expectation, the lighting effect of the captured image is not ideal, therefore, the demand of the user for re-lighting processing of the image is met.

[0211] Scenario two

[0212] In this scenario, for the acquisition process of the initial image, the initial image can be a preview image requested by the camera application to the camera, for example, after the mobile phone detects the click operation of the user on the lighting control, the preview image to be displayed is first lighted, and then the preview image after the lighting processing is displayed in the preview area of the photographing preview interface, which corresponds to the above-mentioned Figure 5 and Figure 6 The application scenario diagram for triggering the lighting processing of the preview image is schematically given. The process of lighting the preview image can be performed by the digital signal processor. Specifically, the key point extraction module, the three-dimensional model construction module, the light distribution determination module and the image fusion module are arranged in the digital signal processor. As shown in Figure 14a After the mobile phone detects the application start triggering operation (i.e. the click operation on the application icon, that is, the user operation 1) of the camera application, the camera application sends an image preview request to the camera service in the application framework layer in response to the user operation; the camera service sends an image acquisition request to the camera HAL in the hardware abstraction layer after receiving the image preview request; the camera HAL sends a camera driver instruction to the camera driver in the kernel layer after receiving the image acquisition request; the camera driver sends a camera control instruction to the camera to control the camera to collect image data in real time and acquire the preview image to be displayed after receiving the camera driver instruction.

[0213] That is, the camera application in the application layer controls the camera driver in the kernel layer through the camera service in the application framework layer and the camera HAL in the hardware abstraction layer, so as to control the camera to take a photo. That is, the camera application instructs the camera driver in the kernel layer through the camera service in the application framework layer and the camera HAL in the hardware abstraction layer, to drive the camera of the electronic device to perform image acquisition processing on the current photographed region, and obtain a preview image.

[0214] After the camera collects the preview image in real time, the camera sends the preview image to the camera driver; the camera driver sends the preview image to the camera HAL; the camera HAL sends the preview image to the camera service; the camera service sends the preview image to the camera application; the preview area of the photo preview interface of the camera application displays the preview image; wherein, when the preview area displays the preview image, the preview image is also refreshed in real time. The refresh process of the preview image can refer to the prior art, which will not be described here.

[0215] Next, after the mobile phone detects the image lighting trigger operation of the user (that is, the click operation on the lighting control, that is, the user operation 2), the mobile phone determines the light source parameter information in response to the user operation; the camera application sends the light source parameter information to the camera service in the application framework layer; the camera service sends the light source parameter information to the camera HAL in the hardware abstraction layer, and the camera HAL sends the light source parameter information to the light distribution determination module in the digital signal processor; and the camera HAL sends the preview image to be displayed to the key point extraction module and the image fusion module in the digital signal processor. The key point extraction module performs key point extraction processing based on the preview image to obtain key point spatial distribution information of the target face; and sends the key point spatial distribution information to the stereoscopic model construction module. The stereoscopic model construction module performs three-dimensional reconstruction processing based on the key point spatial distribution information to obtain a face stereoscopic model of the target face; and sends the face stereoscopic model to the light distribution determination module. The light distribution determination module performs simulated lighting processing based on the light source parameter information and the face stereoscopic model to obtain first light spatial distribution information and second light spatial distribution information; and sends the first light spatial distribution information and the second light spatial distribution information to the image fusion module. The image fusion module performs image fusion processing based on the preview image, the first light spatial distribution information and the second light spatial distribution information to obtain a first lighted image.

[0216] The image fusion module sends the first lighted image to the camera HAL in the hardware abstraction layer; the camera HAL sends the first lighted image to the camera service; the camera service sends the first lighted image to the camera application; the preview area of the photo preview interface of the camera application displays the first lighted image; the user can confirm whether the lighting effect of the first lighted image meets the expectation, to decide whether to trigger the image shooting process through the photo control.

[0217] Furthermore, if the user confirms that the lighting effect of the first illuminated image meets expectations, they can trigger the image capture process via the camera control. For example... Figure 14b As shown, after the mobile phone detects the user's photo-taking trigger operation (i.e., the click operation on the camera control, i.e., user operation 3), in response to the user operation, the camera application sends an image capture request to the camera service in the application framework layer; after receiving the image capture request, the camera service sends an image acquisition request to the camera HAL in the hardware abstraction layer; after receiving the image acquisition request, the camera HAL sends a camera driver instruction to the camera driver in the kernel layer; after receiving the camera driver instruction, the camera driver sends a camera control instruction to the camera to control the camera to acquire image data in real time and obtain the captured image.

[0218] After capturing an image in real time, the camera sends the captured image to the camera driver; the camera driver then sends the captured image to the camera HAL (Hydraulic Alignment). Since the captured image needs to be illuminated, the camera HAL can directly send the captured image to the digital signal processor (DSP). Furthermore, the camera HAL sends light source parameter information from the camera application to the DSP. The DSP performs illumination processing on the captured image based on the light source parameter information to obtain a second illuminated image. The process of determining the second illuminated image can refer to the specific implementation process of generating the first illuminated image described above, and will not be repeated here.

[0219] The digital signal processor sends a second lighting image to the camera HAL, which in turn sends the second lighting image to the camera service. The camera service then stores the second lighting image in a designated location, allowing it to be viewed through a gallery application.

[0220] In this way, when the shooting environment of the electronic device is an outdoor scene with relatively dark light or an indoor environment with relatively dark light, a preview image (i.e., an original image captured by the camera in real time) is first presented in the preview area of the photograph preview interface of the camera application, at which time the user can directly trigger light processing of the preview image based on the simulated light source in the photograph preview interface of the camera application according to actual needs, and then present the target light image corresponding to the preview image (i.e., the image after light processing of the preview image to be displayed) in the photograph preview interface of the camera application, so that the user can confirm whether the light effect of the target light image displayed in the preview area meets expectations in a timely manner during the photograph preview process, and the user can click the photographing control to trigger storage of the target light image corresponding to the captured image without the need to separately store the captured image without light processing. In addition, after photographing is completed, there is no need to open the gallery application or the third-party image editing application again to perform image light processing. In addition, the image light processing process is set in the digital signal processor, and the processing capability of the digital signal processor for image signals is used to improve the generation efficiency of the target light image.

[0221] It can be understood that the preview image can be first sent to the camera application by the camera HAL and the camera service, and then sent to the digital signal processor by the target application through the camera service and the camera HAL. However, after detecting the triggering operation of the user on the light control and before detecting the triggering operation of the user on the cancel light control, the preview image before light processing is not displayed, and the preview image after light processing is directly displayed. Therefore, in order to simplify the transmission path of the preview image, after receiving the preview image to be displayed transmitted by the camera driver, the camera HAL can directly send the preview image to be displayed to the digital signal processor for image light processing to obtain the first light image, thereby omitting the process of sequentially sending the preview image to be displayed to the camera service and the camera application, and then sequentially sending the preview image to be displayed to the camera service and the camera HAL by the camera application. In addition, in the case where the light source parameter information does not change, the camera application does not need to repeatedly and sequentially send the light source parameter information to the camera service and the camera HAL.

[0222] Further, if the user confirms that the light effect of the first light image does not meet expectations, the light effect of the preview image can be adjusted by a specified control until an adjusted light image that meets the expectations of the user is obtained. Corresponding to the above Figure 7 and Figure 8 An application scenario diagram for triggering re-light processing of a preview image is schematically shown. As Figure 14cAs shown, after detecting the click operation of the user on the light control, and without detecting the click operation of the user on the photographing control, the camera continuously returns the preview image to be displayed to the camera HAL through the camera driver, and the camera HAL sends the preview image to be displayed to the digital signal processor after receiving the preview image to be displayed, and the digital signal processor performs image light processing on the preview image to obtain a first light image. The camera HAL returns the first light image to the camera application through the camera service, and the camera application displays the first light image in the preview area.

[0223] Next, after the mobile phone detects the re-lighting trigger operation of the user (i.e., the click operation on the light control or the human-computer interaction control, that is, the user operation 4), the mobile phone determines the adjusted light source parameter information in response to the user operation. The target application sends the adjusted light source parameter information to the camera HAL through the camera service, and the camera HAL sends the adjusted light source parameter information to the digital signal processor. Moreover, since the light effect of the preview image to be displayed needs to be adjusted based on the adjusted light source parameter information, the camera HAL sends the preview image to the digital signal processor. The digital signal processor performs light processing on the preview image to be displayed based on the adjusted light source parameter information to obtain an adjusted light image, and sends the adjusted light image to the camera HAL. The camera HAL sends the adjusted light image to the camera service, and the camera service sends the adjusted light image to the camera application. The preview area of the photographing preview interface of the camera application displays the adjusted light image. The user can confirm whether the light effect of the adjusted light image meets the expectation to decide whether to trigger the image photographing process through the photographing control.

[0224] The adjusted light source parameter information can refer to the specific implementation process described above, and the determination process of the adjusted light image can refer to the specific implementation process of generating the first light image described above, which will not be described here.

[0225] Scenario three

[0226] In this scenario, for the initial image acquisition process, the initial image can also be a historical image photographed by the user from the gallery application. Corresponding to the above Figure 9 An application scenario diagram for triggering light processing on a gallery image is schematically given. For example, the user selects an unlight-processed image as a gallery image, and the process of light processing on the gallery image can be performed by the digital signal processor. Specifically, the key point extraction module, the three-dimensional model construction module, the light distribution determination module, and the image fusion module are arranged in the digital signal processor. For example, Figure 14dAs shown, after the mobile phone detects the user's photographing trigger operation (i.e. the click operation on the photographing control, also known as user operation 5), the camera application sends an image shooting request to the camera service in response to the user operation; the camera service sends an image collection request to the camera HAL after receiving the image shooting request; the camera HAL sends a camera driver instruction to the camera driver after receiving the image collection request; the camera driver sends a camera control instruction to the camera to control the camera to collect image data in real time and obtain a shooting image after receiving the camera driver instruction.

[0227] After the camera collects the shooting image in real time, the camera sends the shooting image to the camera driver; the camera driver sends the shooting image to the camera HAL; the camera HAL sends the shooting image to the camera service; and the shooting image data is stored in a specified location, so that the shooting image can be viewed by the gallery application and selected as a gallery image to be processed.

[0228] Then, in the case that the user triggers the opening of the gallery application, the user can select a gallery image to be processed from a plurality of gallery images. After the mobile phone detects the user's image lighting trigger operation (i.e. the click operation on the lighting control, also known as user operation 6), the light source parameter information is obtained in response to the user operation; the gallery application sends the gallery image to be processed and the light source parameter information to the image processing service in the application framework layer; the image processing service sends the gallery image to be processed and the light source parameter information to the camera HAL in the hardware abstraction layer; then, the camera HAL sends the gallery image to be processed to the key point extraction module and the image fusion module in the digital signal processor, and sends the light source parameter information to the light distribution determination module. The key point extraction module performs key point extraction processing based on the gallery image to obtain key point spatial distribution information of the target face; and sends the key point spatial distribution information to the three-dimensional model construction module. The three-dimensional model construction module performs three-dimensional reconstruction processing based on the key point spatial distribution information to obtain a three-dimensional face model of the target face; and sends the three-dimensional face model to the light distribution determination module. The light distribution determination module performs simulated lighting processing based on the light source parameter information and the three-dimensional face model to obtain first light spatial distribution information and second light spatial distribution information; and sends the first light spatial distribution information and the second light spatial distribution information to the image fusion module. The image fusion module performs image fusion processing based on the gallery image, the first light spatial distribution information and the second light spatial distribution information to obtain a third lighting image.

[0229] The image fusion module sends the third lighted image to the camera HAL in the hardware abstraction layer; the camera HAL sends the third lighted image to the image processing service; the image processing service sends the third lighted image to the gallery application; and the gallery application displays the third lighted image on the image editing interface. If the currently displayed third lighted image meets the user's expectations, the user can trigger saving of the third lighted image. If the currently displayed target lighted image does not meet the user's expectations, the user can again set the image light parameter and trigger regeneration of the target lighted image; in addition, the user can also trigger recapturing of a new gallery image and regeneration of a target lighted image corresponding to the new gallery image. The specific implementation process can refer to the above process, and will not be described here again.

[0230] Next, after detecting the user's confirmation trigger operation (i.e., a click operation on the save control, i.e., user operation 7), the phone responds to the user operation, and the gallery application sends a lighted image storage request to the image processing service. The image processing service stores the third lighted image in a specified location, so that the third lighted image can be viewed through the gallery application.

[0231] In this way, by adding an image light control in the gallery application, after image capture is complete, if the user enters the gallery application to view the captured images, for unlighted gallery images, or the user finds that the lighting effect of the lighted image does not meet expectations, the user can directly trigger light processing of the captured image at any time in the gallery application to obtain a target lighted image that meets the user's expectations, ensuring that the user's demand for image light processing is met in different application scenarios. Moreover, triggering image light processing directly in the gallery application also does not require opening a third-party photo editing application for image light processing. In addition, the image light processing process is set in the digital signal processor, and the processing capability of the digital signal processor for image signals is used to improve the generation efficiency of the target lighted image.

[0232] It can be understood that the user can choose either of the above two implementation manners according to the actual scene, providing the user with more diverse implementation manners for triggering image light processing.

[0233] In addition, the preview image refers to an image displayed in real time on the photograph preview interface of the photographing application before the user clicks the photographing control; and the captured image refers to image data captured by the camera when the user clicks the photographing control. Generally, after the camera application is opened and before the user clicks the photographing control, the image preview stage is entered, and the preview image is displayed on the photograph preview interface of the photographing application, and the preview image is constantly refreshed. When the user clicks the photographing control, the image capture stage is entered, and the captured image captured by the camera is controlled. Next, after the gallery application is opened, the image viewing stage is entered, and the historical captured image is displayed on the display interface of the gallery application.

[0234] In the embodiments provided in the present application, before the user clicks the photographing control, that is, in the image preview stage, the preview image can be lighted based on the simulated light source by clicking the lighting control, and the target lighting image corresponding to the preview image is displayed on the photographing preview interface of the camera application, at this time the user can identify whether the lighting effect of the displayed target lighting image meets the expectation; in the case that the user confirms that the image lighting effect meets the expectation, the target lighting image corresponding to the photographed image can be stored by clicking the photographing control. Specifically, in the case that the click operation of the first lighting control (that is, the lighting control on the display interface of the camera application) of the user is detected, the preview image to be displayed is lighted first to obtain a first lighting image; then the first lighting image corresponding to the preview image is displayed on the photographing preview interface of the photographing application. Then, when the user clicks the photographing control, the photographed image captured by the camera is controlled; the photographed image is lighted to obtain a second lighting image, and the second lighting image corresponding to the photographed image is stored, so that the second lighting image can be viewed in the gallery application.

[0235] In addition, after the gallery application is opened, the image viewing stage is entered, the gallery image (such as historical photographed image data) can also be lighted based on the simulated light source by clicking the lighting control, and the target lighting image corresponding to the gallery image is displayed on the display interface of the gallery application, at this time the user can identify whether the lighting effect of the displayed target lighting image meets the expectation; in the case that the user confirms that the image lighting effect meets the expectation, the target lighting image corresponding to the gallery image can be stored by clicking the save control. Specifically, in the case that the click operation of the second lighting control (that is, the lighting control on the display interface of the gallery application) of the user is detected, the gallery image selected by the user is lighted first to obtain a third lighting image; then the third lighting image corresponding to the gallery image is displayed on the display interface of the gallery application. Then, when the user clicks the save control, the third lighting image corresponding to the gallery image is stored, so that the third lighting image can be viewed in the gallery application.

[0236] It should be noted that the process of image lighting processing can also be performed by an application processor, specifically by a function module in an application framework layer, or by a function module in a hardware abstraction layer. Specifically, in the case of image lighting processing by a function module in the application framework layer, the application framework layer further includes an image processing service, which includes the key point extraction module, the stereo model construction module, the light distribution determination module, and the image fusion module. In response to an image lighting trigger operation, the phone sends the initial image to be processed and the light source parameter information to the image processing service. The image processing service performs image lighting processing based on the initial image and the light source parameter information to obtain a target lighting image. Then, the image processing service returns the target lighting image to the camera application, and the preview area of the camera application's photograph preview interface displays the target lighting image. In the case of image lighting processing by a function module in the hardware abstraction layer, the camera HAL includes the key point extraction module, the stereo model construction module, the light distribution determination module, and the image fusion module. In response to an image lighting trigger operation, the phone sends the initial image to be processed and the light source parameter information to the camera application. The camera service sends the initial image to be processed and the light source parameter information to the camera HAL. The camera HAL performs image lighting processing based on the initial image and the light source parameter information to obtain a target lighting image. Then, the camera HAL returns the target lighting image to the camera service, and the camera service returns the target lighting image to the camera application, and the preview area of the camera application's photograph preview interface displays the target lighting image. The generation process of the target lighting image can refer to the specific implementation process of S701 to S708, which will not be described here.

[0237] Scenario Four

[0238] In this scenario, the target shooting object is a target face. In one case, considering that the initial image not only includes the image region of the target face, but also may contain the image region of the target human body corresponding to the target face, in order to ensure the coordination of the lighting effects of the target face and the target human body. Taking an initial image containing a target face and a target human body corresponding to the target face as an example, as shown in FIG. 8A, the specific implementation process of generating a target lighting image corresponding to the initial image can include: Figure 15

[0239] The image segmentation module performs region segmentation processing on the initial image to obtain a first sub-image containing the target face and a second sub-image containing the target human body. For example, a pre-trained image segmentation model can be used to perform image region segmentation of the face and the human body on the initial image to obtain an image region containing the target face (i.e., the first sub-image) and an image region containing the target human body (i.e., the second sub-image).

[0240] ​The key point extraction module performs key point extraction processing on the first sub-image to obtain key point spatial distribution information of the target face. The determination process of the key point spatial distribution information can refer to the specific implementation process in the above Figure 10 , and will not be described here.

[0241] The three-dimensional model construction module performs three-dimensional reconstruction processing based on the key point spatial distribution information of the target face to obtain a face three-dimensional model of the target face. The construction process of the face three-dimensional model can refer to the specific implementation process in the above Figure 10 , and will not be described here.

[0242] The light distribution determination module generates first light spatial distribution information corresponding to highlight reflection and second light spatial distribution information corresponding to diffuse reflection of the target face based on the light source parameter information of the simulated lighting light source and the face three-dimensional model. The determination process of the first light spatial distribution information and the second light spatial distribution information can refer to the specific implementation process in the above Figure 10 , and will not be described here.

[0243] The light distribution determination module performs planar projection based on the first light spatial distribution information to obtain first light planar distribution information corresponding to the target face. Specifically, a corresponding relationship between each first pixel value in the first light spatial distribution information and each pixel point in the first sub-image is established to obtain the first light planar distribution information.

[0244] The light distribution determination module performs planar projection based on the second light spatial distribution information to obtain second light planar distribution information corresponding to the target face. Specifically, a corresponding relationship between each second pixel value in the second light spatial distribution information and each pixel point in the first sub-image is established to obtain the second light planar distribution information.

[0245] The light distribution prediction module performs human body light distribution prediction based on the second sub-image, the first light planar distribution information, and the light source parameter information to obtain third light planar distribution information of the target human body. The third light planar distribution information includes third pixel values of each pixel point in the second sub-image, and each third pixel value is used to represent the highlight reflection light color of a certain pixel point of the target human body.

[0246] Exemplarily, the first light distribution prediction model is used to predict the highlight reflection light distribution of the target human body based on the highlight reflection light distribution of the target face. In addition, in the process of human light distribution prediction, the light source parameter information can also be used as a reference, and the coordinate information of each pixel point in the initial image can represent the relative position relationship between the target human body and the target face. Therefore, the first light plane distribution information, the light source parameter information and the coordinate information of each pixel point in the initial image can be input into the first light distribution prediction model for light prediction to obtain the third light plane distribution information of the target human body.

[0247] The light distribution prediction module predicts the human light distribution based on the second sub-image, the second light plane distribution information and the light source parameter information to obtain the fourth light plane distribution information of the target human body. The fourth light plane distribution information includes fourth pixel values of each pixel point in the second sub-image, and each fourth pixel value is used to represent the diffuse reflection light color of a pixel point of the target human body.

[0248] Exemplarily, the second light distribution prediction model is used to predict the human light distribution based on the relative position relationship between the second sub-image and the first sub-image, the second light plane distribution information and the light source parameter information to obtain the fourth light plane distribution information of the target human body. The second light distribution prediction model is used to predict the diffuse reflection light distribution of the target human body based on the diffuse reflection light distribution of the target face. In addition, in the process of human light distribution prediction, the light source parameter information can also be used as a reference, and the coordinate information of each pixel point in the initial image can represent the relative position relationship between the target human body and the target face. Therefore, the second light plane distribution information, the light source parameter information and the coordinate information of each pixel point in the initial image can be input into the second light distribution prediction model for light prediction to obtain the fourth light plane distribution information of the target human body.

[0249] The light distribution combination module performs portrait synthesis processing based on the first light plane distribution information and the third light plane distribution information to obtain the first overall light distribution information. The first light plane distribution information is used to represent the highlight reflection light distribution of the target face, and the third light plane distribution information is used to represent the highlight reflection light distribution of the target human body. Therefore, based on the relative position relationship between the target face and the target human body in the initial image, the first light plane distribution information and the third light plane distribution information are spliced to obtain the first overall light distribution information. The first overall light distribution information is used to represent the highlight reflection light distribution of the portrait of the target user.

[0250] The light distribution combination module performs portrait synthesis processing based on the second light plane distribution information and the fourth light plane distribution information to obtain second overall light distribution information. The second light plane distribution information is used to represent the diffuse reflection light distribution of the target face, and the fourth light plane distribution information is used to represent the diffuse reflection light distribution of the target body. Therefore, based on the relative position relationship between the target face and the target body in the initial image, the second light plane distribution information and the fourth light plane distribution information are spliced to obtain the second overall light distribution information. The second overall light distribution information is used to represent the diffuse reflection light distribution of the target user.

[0251] The image fusion module performs image fusion processing based on the initial image, the first overall light distribution information and the second overall light distribution information to obtain a target lighting image. Specifically, first, the first pixel value of the target pixel point and the third pixel value of the body pixel point in the initial image are determined according to the first overall light distribution information; and the second pixel value of the target pixel point and the fourth pixel value of the body pixel point in the initial image are determined according to the second overall light distribution information; the target pixel point is any pixel point on the face three-dimensional model, and the body pixel point is any pixel point in the image region where the target body is located; then, the original pixel value, the first pixel value and the second pixel value of the target pixel point are weighted and summed to obtain the fusion pixel value of the target pixel point; and the original pixel value, the third pixel value and the fourth pixel value of the body pixel point are weighted and summed to obtain the fusion pixel value of the body pixel point; wherein the original pixel value is used to represent the pixel color of the initial image; next, the target lighting image is generated according to the original pixel value of each non-target pixel point, the fusion pixel value of each target pixel point and the fusion pixel value of each body pixel point in the initial image; the non-target pixel point is a pixel point in the initial image that is not processed by lighting.

[0252] In another case, considering that the initial image not only includes the image region of the target face, but also can contain the image region of the background shooting object of the target face, taking an initial image including a target face and a background shooting object as an example, as shown in Figure 16 The specific implementation process of generating a target lighting image corresponding to the initial image can include:

[0253] The image segmentation module performs region segmentation processing on the initial image to obtain a third sub-image containing the target face and a fourth sub-image containing the background shooting object. Illustratively, a pre-trained image segmentation model can be used to perform image region segmentation of the face and the background of the initial image to obtain an image region containing the target face (i.e., the third sub-image) and an image region containing the background shooting object (i.e., the fourth sub-image).

[0254] The key point extraction module performs key point extraction processing on the third sub-image to obtain key point spatial distribution information of the target face. The determination process of the key point spatial distribution information can refer to the specific implementation process in the above Figure 10 , and will not be described here.

[0255] The three-dimensional model construction module performs three-dimensional reconstruction processing based on the key point spatial distribution information of the target face to obtain a face three-dimensional model of the target face. The construction process of the face three-dimensional model can refer to the specific implementation process in the above Figure 10 , and will not be described here.

[0256] The light distribution determination module generates first light spatial distribution information corresponding to highlight reflection and second light spatial distribution information corresponding to diffuse reflection of the target face based on the light source parameter information of the simulated lighting light source and the face three-dimensional model. The determination process of the first light spatial distribution information and the second light spatial distribution information can refer to the specific implementation process in the above Figure 10 , and will not be described here.

[0257] The light distribution determination module performs planar projection based on the first light spatial distribution information to obtain first light planar distribution information corresponding to the target face. Specifically, a corresponding relationship between each first pixel value in the first light spatial distribution information and each pixel point in the third sub-image is established to obtain the first light planar distribution information.

[0258] The light distribution determination module performs planar projection based on the second light spatial distribution information to obtain second light planar distribution information corresponding to the target face. Specifically, a corresponding relationship between each second pixel value in the second light spatial distribution information and each pixel point in the third sub-image is established to obtain the second light planar distribution information.

[0259] The light distribution prediction module performs background light distribution prediction based on the fourth sub-image, the first light planar distribution information, and the light source parameter information to obtain fifth light planar distribution information of the background shooting object. The fifth light planar distribution information includes fifth pixel values of each pixel point in the fourth sub-image, and each fifth pixel value is used to represent the highlight reflection light color of a pixel point of the background shooting object.

[0260] Exemplarily, the fourth sub-image, the second light plane distribution information and the light source parameter information are input into a third light distribution prediction model to predict the background light distribution based on the relative position relationship between the target face and the background subject, to obtain fifth light plane distribution information of the background subject. The third light distribution prediction model is used to predict the highlight reflection light distribution of the background subject based on the highlight reflection light distribution of the target face. In the model training stage, the third light distribution prediction model and the training sample set used by the first light distribution prediction model can be different. In addition, in the process of predicting the background light distribution, the light source parameter information can also be used as a reference, and the coordinate information of each pixel point in the initial image can represent the relative position relationship between the target face and the background subject. Therefore, the first light plane distribution information, the light source parameter information and the coordinate information of each pixel point in the initial image can be input into the third light distribution prediction model to predict the light distribution, to obtain the fifth light plane distribution information of the background subject.

[0261] The light distribution prediction module predicts the background light distribution based on the fourth sub-image, the second light plane distribution information and the light source parameter information, to obtain sixth light plane distribution information of the background subject. The sixth light plane distribution information includes sixth pixel values of each pixel point in the fourth sub-image, and each sixth pixel value is used to represent the diffuse reflection light color of a pixel point of the background subject.

[0262] Exemplarily, the fourth sub-image, the second light plane distribution information and the light source parameter information are input into a third light distribution prediction model to predict the background light distribution based on the relative position relationship between the target face and the background subject, to obtain fifth light plane distribution information of the background subject. The third light distribution prediction model is used to predict the highlight reflection light distribution of the background subject based on the highlight reflection light distribution of the target face. In the model training stage, the third light distribution prediction model and the training sample set used by the first light distribution prediction model can be different. In addition, in the process of predicting the background light distribution, the light source parameter information can also be used as a reference, and the coordinate information of each pixel point in the initial image can represent the relative position relationship between the target face and the background subject. Therefore, the first light plane distribution information, the light source parameter information and the coordinate information of each pixel point in the initial image can be input into the third light distribution prediction model to predict the light distribution, to obtain the fifth light plane distribution information of the background subject.

[0263] The light distribution combination module performs foreground and background synthesis processing based on the first light plane distribution information and the fifth light plane distribution information to obtain first panoramic light distribution information. The first light plane distribution information is used to represent the highlight reflection light distribution of the target face, and the fifth light plane distribution information is used to represent the highlight reflection light distribution of the background shooting object. Therefore, based on the relative position relationship between the target face and the background shooting object in the initial image, the first light plane distribution information and the fifth light plane distribution information are spliced to obtain the first panoramic light distribution information. The first panoramic light distribution information is used to represent the highlight reflection light distribution of the panorama.

[0264] The light distribution combination module performs foreground and background synthesis processing based on the second light plane distribution information and the sixth light plane distribution information to obtain second panoramic light distribution information. The second light plane distribution information is used to represent the diffuse reflection light distribution of the target face, and the sixth light plane distribution information is used to represent the diffuse reflection light distribution of the background shooting object. Therefore, based on the relative position relationship between the target face and the background shooting object in the initial image, the second light plane distribution information and the sixth light plane distribution information are spliced to obtain the second panoramic light distribution information. The second panoramic light distribution information is used to represent the diffuse reflection light distribution of the panorama.

[0265] The image fusion module performs image fusion processing based on the initial image, the first panoramic light distribution information and the second panoramic light distribution information to obtain a target lighting image. Specifically, first, according to the first panoramic light distribution information, the first pixel value of the target pixel point and the fifth pixel value of the background pixel point in the initial image are determined; and according to the second panoramic light distribution information, the second pixel value of the target pixel point and the sixth pixel value of the background pixel point in the initial image are determined; the target pixel point is any pixel point on the face model, and the background pixel point is any pixel point in the image region of the background shooting object; then, the original pixel value, the first pixel value and the second pixel value of the target pixel point are weighted and summed to obtain the fusion pixel value of the target pixel point; and the original pixel value, the fifth pixel value and the sixth pixel value of the background pixel point are weighted and summed to obtain the fusion pixel value of the background pixel point; wherein the original pixel value is used to represent the pixel color of the initial image; next, according to the original pixel value of each non-target pixel point in the initial image, the fusion pixel value of each target pixel point and the fusion pixel value of each background pixel point, a target lighting image is generated; the non-target pixel point is a pixel point in the initial image that is not processed by lighting.

[0266] It can be understood that if the initial image includes a target face, a target human body corresponding to the target face, and a background shooting object, the process of generating the target lighted image can refer to the specific implementation process given in the above scenario four and scenario five, and details are not repeated here.

[0267] In addition, if the initial image contains two or more target faces, the plurality of target faces can be segmented from the initial image first, and the key point spatial distribution information corresponding to each target face is determined; then for each target face, a corresponding face three-dimensional model is constructed; then for each face three-dimensional model, corresponding light space distribution information is generated; and then based on the initial image and the light space distribution information corresponding to each target face, image fusion processing is performed to obtain a target lighted image. The specific implementation process can refer to the above embodiments, and details are not repeated here.

[0268] It should be noted that the image processing method provided by the embodiments of the present application is applicable to light processing of video data; for example, part or all of the image frames in the video data can be light processed to obtain light processed video data; wherein the specific implementation process of light processing each image frame can refer to the above detailed description, and details are not repeated here.

[0269] The embodiment also provides an electronic device, which includes one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the related method steps to implement the image processing method in the above embodiments.

[0270] The embodiment also provides a computer storage medium, which stores computer instructions, and when the computer instructions run on an electronic device, the electronic device executes the related method steps to implement the image processing method in the above embodiments.

[0271] The embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the related steps to implement the image processing method in the above embodiments.

[0272] In addition, the embodiments of the present application also provide a device, which can be a chip, a component or a module. The device can include a processor and a memory connected to each other; wherein the memory is used to store computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory to make the chip execute the image processing method in the above method embodiments.

[0273] In addition, the embodiments of the present application also provide a chip, which can include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuits communicate with each other through internal connection paths, and the processing circuits perform the above-mentioned related method steps to realize the image processing method in the above-mentioned embodiments to control the receiving pins to receive signals and control the sending pins to send signals.

[0274] Among them, the electronic device (such as a mobile phone and the like), the computer storage medium, the computer program product, the device or the chip provided by the embodiment are used to execute the corresponding method provided above, so the beneficial effects that can be achieved are referred to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0275] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the above-mentioned division of functional modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0276] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0277] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, including: The first preview interface of the target application is displayed; the target application has an image acquisition function, and the first preview interface includes a first control. In response to a first operation on the first control, a second preview interface of the target application is displayed; the second preview interface includes a second preview image, which is obtained by illuminating the target object in the first preview image with a simulated light source, and the first preview image is a frame preview image captured by the electronic device.

2. The method according to claim 1, characterized in that, Before displaying the second preview interface of the target application, the following is also included: Display a first settings interface, and in response to a second operation performed on the first settings interface, determine the first parameter information of the simulated light source; The target object in the first preview image is illuminated based on the first parameter information to obtain the second preview image.

3. The method according to claim 2, characterized in that, The second preview interface also includes a second control; After displaying the second preview interface of the target application, the following is also included: In response to a third operation on the second control, a first captured image is acquired, and the target object in the first captured image is illuminated according to the first parameter information to obtain a second captured image; Store the second captured image.

4. The method according to claim 2, characterized in that, The second preview interface also includes a third control; After displaying the second preview interface of the target application, the following is also included: In response to a fourth operation on the third control, second parameter information of the simulated light source is determined; The target object in the third preview image is illuminated according to the second parameter information to obtain the fourth preview image; the third preview image is a frame preview image captured by the electronic device. The fourth preview image is displayed in the second preview interface.

5. The method according to claim 1, characterized in that, Also includes: The first interface of the gallery application is displayed; the first interface includes a first image and a fourth control, wherein the first image is the image selected by the user; In response to a fifth operation on the fourth control, a second settings interface is displayed; In response to the sixth operation performed on the second setting interface, the third parameter information of the simulated light source is determined; The target object in the first image is illuminated according to the third parameter information to obtain a second image, which is then displayed on the second interface of the image library application.

6. The method according to claim 2, characterized in that, The step of applying lighting processing to the target object in the first preview image based on the first parameter information to obtain the second preview image includes: The first preview image is subjected to key point extraction processing to obtain the key point distribution information of the target object in the first preview image; Based on the key point distribution information, a three-dimensional reconstruction process is performed to obtain a three-dimensional model of the target object. Based on the first parameter information, simulate lighting processing is performed on the three-dimensional model to obtain the target illumination distribution information corresponding to the three-dimensional model; The second preview image is obtained by performing image fusion processing based on the first preview image and the target illumination distribution information.

7. The method according to claim 6, characterized in that, The step of performing three-dimensional reconstruction processing based on the key point distribution information to obtain a three-dimensional model of the target object includes: From the multiple key points in the key point distribution information, select the first point and the second point; Using the domain growth method, starting from the first point and ending at the second point, the current growth point is determined sequentially from the plurality of key points; Establish the connection relationship between the current growth point and the domain growth point of the current growth point to obtain the three-dimensional model of the target object.

8. The method according to claim 6, characterized in that, The step of simulating lighting on the three-dimensional model based on the first parameter information to obtain the target illumination distribution information corresponding to the three-dimensional model includes: Based on the first parameter information, determine the target parameter information for each unit surface in the three-dimensional model; Based on the target parameter information, the specular reflection illumination is calculated to obtain the first illumination distribution information; Based on the target parameter information, diffuse reflection illumination is calculated to obtain the second illumination distribution information; The first illumination distribution information and the second illumination distribution information are determined as the target illumination distribution information.

9. The method according to claim 8, characterized in that, The step of calculating the specular reflectance illumination based on the target parameter information to obtain the first illumination distribution information includes: Based on the target parameter information, specular reflection illumination is calculated to obtain first initial distribution information; the first initial distribution information is then smoothed to obtain first illumination distribution information. The step of calculating diffuse reflection illumination based on the target parameter information to obtain second illumination distribution information includes: Diffuse reflection illumination is calculated based on the target parameter information to obtain second initial distribution information; the second initial distribution information is then smoothed to obtain second illumination distribution information.

10. The method according to claim 6, characterized in that, The target illumination distribution information includes first illumination distribution information and second illumination distribution information; The step of performing image fusion processing based on the first preview image and the target illumination distribution information to obtain the second preview image includes: Based on the first illumination distribution information, a first pixel value of the target pixel is determined; and based on the second illumination distribution information, a second pixel value of the target pixel is determined; the target pixel is any pixel on the three-dimensional model. The original pixel value, the first pixel value, and the second pixel value of the target pixel are weighted and summed to obtain the fused pixel value; wherein, the original pixel value is used to characterize the pixel color of the first preview image; A second preview image is generated based on the original pixel values ​​of each non-target pixel in the first preview image and the fused pixel values ​​of each target pixel; the non-target pixels are the pixels in the first preview image that have not been illuminated.

11. The method according to claim 6, characterized in that, The target object is a target human face, and the first preview image also includes the target human body corresponding to the target human face; After performing simulated lighting processing on the three-dimensional model based on the first parameter information to obtain the target illumination distribution information map corresponding to the three-dimensional model, the method further includes: Human body illumination distribution is predicted based on the target illumination distribution information to obtain the human body illumination distribution information of the target human body; The step of performing image fusion processing based on the first preview image and the target illumination distribution information to obtain the second preview image includes: The target illumination distribution information and the human body illumination distribution information are combined to obtain overall illumination distribution information; the first preview image and the overall illumination distribution information are then combined to obtain a second preview image.

12. The method according to claim 6, characterized in that, The target object is a target human face, and the first preview image also includes a background object that has a certain positional relationship with the target human face; After performing simulated lighting processing on the three-dimensional model based on the first parameter information to obtain the target illumination distribution information corresponding to the three-dimensional model, the method further includes: Based on the target illumination distribution information and the positional relationship, the background illumination distribution is predicted to obtain the background illumination distribution information of the background object being photographed. The step of performing image fusion processing based on the first preview image and the target illumination distribution information to obtain the second preview image includes: The target illumination distribution information and the background illumination distribution information are combined to obtain panoramic illumination distribution information; the first preview image and the panoramic illumination distribution information are then fused to obtain a second preview image.

13. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the image processing method as described in any one of claims 1 to 12.

14. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, the electronic device causes the electronic device to perform the image processing method as described in any one of claims 1 to 12.

15. A chip used in an electronic device, the chip comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 12.

Images