Image processing method and apparatus, electronic device, and medium

Abstract

Provided are an image processing method and apparatus, an electronic device, and a medium. The method comprises: calculating a first normal map and a lighting reflectivity weight distribution diagram of a foreground image in a two-dimensional image; on the basis of the first normal map, the lighting reflectivity weight distribution diagram, and first light source attribute information, performing lighting rendering processing on a first image, wherein the first image comprises a two-dimensional image or a three-dimensional image corresponding to the two-dimensional image; and displaying the rendered first image.

Description

Image processing method, device, electronic device and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311062129.0, filed on August 22, 2023, entitled “Image processing method, device, electronic device and medium”, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device, electronic device and medium. Background Art

[0004] With the development of camera technology, more and more users are pursuing the atmosphere of photos when taking pictures.

[0005] For example, in order to enhance the lighting effect of a photo, a lighting effect may be added to the photo. For example, the electronic device may overlay light on the photo to enhance the atmosphere of the photo.

[0006] However, since electronic devices add light effects to photos by superimposing layers, the light effects may weaken or blur the original three-dimensional effect of the photo, resulting in poor display effect of the photo after adding the light effects.

[0007] Summary of the Invention

[0008] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and medium, which can solve the problem of poor display effect of photos after adding light effects.

[0009] An embodiment of the present application provides an image processing method, which includes: calculating a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image; performing light rendering processing on a first image based on the first normal map, the light reflectance weight distribution map, and first light source attribute information, where the first image includes a two-dimensional image or a three-dimensional image corresponding to the two-dimensional image; and displaying the rendered first image.

[0010] An embodiment of the present application also provides an image processing device, which may include: a processing module, a rendering module and a display module; the processing module is used to calculate a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image; the rendering module is used to perform light rendering processing on the first image processed by the processing module based on the first normal map, the light reflectance weight distribution map and the first light source attribute information, the first image including a two-dimensional image or a three-dimensional image corresponding to the two-dimensional image; the display module is used to display the first image rendered by the rendering module.

[0011] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0012] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0014] An embodiment of the present application further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method as described in the first aspect.

[0015] In an embodiment of the present application, a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image can be calculated; based on the first normal map, the light reflectance weight distribution map, and the first light source attribute information, a light rendering process is performed on the first image, the first image including the two-dimensional image or the three-dimensional image corresponding to the two-dimensional image; and the rendered first image is displayed. Through this solution, when a two-dimensional image needs to be light-rendered, since the first normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image are first calculated, and then based on the normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image, the two-dimensional image or the three-dimensional image corresponding to the two-dimensional image is light-rendered, it can be ensured that the rendered image has a good three-dimensional sense, thereby improving the display effect of the image after adding the light effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a flow chart of an image processing method according to an embodiment of the present invention;

[0017] FIG2 is a schematic diagram of a first normal map of a foreground image in a two-dimensional image in the image processing method provided by an embodiment of the present application;

[0018] Figure 3 is a schematic diagram of a neural network model;

[0019] FIG4 is a schematic diagram of a two-dimensional image in an image processing method provided in an embodiment of the application;

[0020] FIG5 is a schematic diagram of a pixel image of a foreground image in a two-dimensional image;

[0021] FIG6(A) is one of the interface diagrams of the application of the image processing method provided in the embodiment of the application;

[0022] FIG6(B) is a second schematic diagram of an interface for applying the image processing method provided in the embodiment of the application;

[0023] FIG6(C) is a third schematic diagram of an interface for applying the image processing method provided in the embodiment of the application;

[0024] FIG6(D) is a fourth schematic diagram of an interface for applying the image processing method provided in an embodiment of the application;

[0025] FIG7 is a second flow chart of the image processing method provided in an embodiment of the present application;

[0026] FIG8 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0027] FIG9 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application;

[0028] FIG10 is a second structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0030] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The terms "at least one" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the objects included. For example, at least one of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "at least two" means two or more, and its meaning is similar to "at least one".

[0032] The following is an explanation of the nouns and terms involved in the description and claims of this application.

[0033] Depth estimation: Performs high-precision depth estimation on a single 2D static image to obtain depth information for each pixel in the 2D image. This depth information can be used to determine the distance of the subject in the 2D image from the camera.

[0034] Normal Map: A bump map. A normal map is a special texture that can extract surface details (such as bumps, grooves, and scratches) of objects in the image.

[0035] obj file: is a 3D model file format. The obj file contains information about the geometry of the 3D object.

[0036] Rendering: Rendering is the process of converting graphic data from a computer program into an image. In other words, it's the process of transforming the designed content into a final rendering or animation using the software itself or auxiliary software. By adding effects like textures during the rendering process, 3D images can be made more realistic and texture-rich.

[0037] The image processing method, device, electronic device, and medium provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0038] The image processing method, device, electronic device, and medium provided in the embodiments of the present application can be applied to scenes where lighting rendering is performed on two-dimensional images to enhance the atmosphere of the two-dimensional images.

[0039] With the development of camera technology, more and more users are pursuing the texture and atmosphere of photos when taking pictures. One of the effective ways to enhance the texture and atmosphere is to add lighting.

[0040] Currently, there are two main approaches to adding lighting effects. The first involves setting up lighting before taking a photo, allowing for immediate capture. This requires specialized fill light equipment, while a simpler approach involves using a flashlight from another phone. Similarly, ambient lighting can be used to adjust the fill light's color to create a more atmospheric effect. However, current lighting methods often require manual setup of the lighting scene, making lighting equipment expensive. Even the simplest fill light requires investment. Furthermore, the setup process can be complex, time-consuming, and labor-intensive, requiring more than just one light source. Furthermore, for devices with built-in flash, since the flash's direction and color cannot be changed, using the flash only provides fill light and cannot add lighting effects.

[0041] Method 2 is post-production photo editing, which is usually done using professional photo editing software. However, since photos are flat, post-production photo editing cannot simulate the real lighting effect, which will cause the three-dimensional effect of the original picture to be eroded when adding lighting effects.

[0042] As such, the process of producing an atmosphere image in the related art is cumbersome and costly, or the display effect of the produced atmosphere image is poor.

[0043] The image processing method provided in the embodiment of the present application can simulate the fill light effect of a real fill light and add light effects to an existing 2D image to ensure that the rendered image has a good three-dimensional sense. For example, it can achieve the addition of 360-degree real light effects to the subject.

[0044] For example, when it is required to perform lighting rendering on a two-dimensional image, the three-dimensional image corresponding to the two-dimensional image is first determined based on the depth estimation information of the two-dimensional image, and then the three-dimensional lighting rendering is performed on the three-dimensional image based on the normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image. Therefore, it can be ensured that the two-dimensional mapping image of the rendered three-dimensional image has a three-dimensional lighting effect rendering effect, that is, the addition of three-dimensional lighting effects to the two-dimensional image is achieved, which can improve the display effect of the image after adding the lighting effect.

[0045] In some embodiments of the present application, since the two-dimensional image is first reconstructed into three dimensions, the three-dimensional image corresponding to the two-dimensional image can be a three-dimensional spatial image, and the depth information of the foreground image in the two-dimensional image is different from the depth information of the background image. Therefore, a light source can be set between the foreground object and the background object in the corresponding three-dimensional image, thereby realizing the addition of a backlight and achieving the effect of adding contour light to the foreground object.

[0046] It can be understood that the foreground object refers to an object in the three-dimensional image that corresponds to the foreground image in the two-dimensional image.

[0047] In some embodiments of the present application, the color temperature or color light of a two-dimensional image may be adjusted by rendering a three-dimensional image.

[0048] In some embodiments of the present application, a light source can be added to a two-dimensional image by rendering a three-dimensional image, such as supporting the addition of multiple light sources, so that the atmosphere of the two-dimensional mapping image obtained after rendering is better.

[0049] The image processing method provided in the embodiment of the present application may be executed by an image processing device, which may be an electronic device or a functional module in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0050] An embodiment of the present application provides an image processing method. FIG1 shows a flow chart of the image processing method provided by the embodiment of the present application. As shown in FIG1 , the image processing method may include the following steps: 101 to 103 .

[0051] Step 101: The electronic device calculates a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image.

[0052] In some embodiments of the present application, the above-mentioned two-dimensional image can be a preview image displayed in a shooting preview interface, or an image in an album of an electronic device, or an image downloaded by the electronic device.

[0053] As you can understand, a normal map is a detail-enhancing map used to create surface relief textures. Its image primarily stores surface normal information. Adding a normal map to the rendering process can enhance the realism and three-dimensionality of lighting.

[0054] In some embodiments of the present application, the first normal map of the foreground image can reflect the direction in which the object in the foreground image receives light. For example, as shown in Figure 2, the first normal map of the foreground image can reflect that the object in the foreground image is backlit, wherein the arrow in Figure 2 is only used to illustrate the direction of light and does not mean that there is an arrow in the actual normal map.

[0055] For example, the first normal map may reflect that an object in the foreground image is backlit or backlit.

[0056] For example, a first normal map may reflect the angle at which objects in the foreground image receive light.

[0057] It can be understood that the first normal map can better simulate the actual lighting effect when shooting a two-dimensional image. For example, the normal map can extract and restore surface details of an object in a two-dimensional image, such as bumps, grooves, and scratches, thereby capturing light as if represented by real geometry.

[0058] For example, if the object in the foreground image of a 2D image is a person, then since the depth estimation information of the 2D image is obtained, the stereoscopic appearance of the face, nose, eyes, and mouth of the person in the 3D image reconstructed based on this depth estimation information may be poor, that is, the detail restoration of the 3D image is poor. Therefore, extracting the discovery map of the foreground image in the 2D image can effectively compensate for the problem of poor detail restoration in the 3D image.

[0059] In some embodiments of the present application, under the same lighting conditions, the greater the light reflectance weight of an image, the higher the brightness of the image.

[0060] In some embodiments of the present application, the lighting system includes ambient light, diffuse light, and specular light. Without adding a light reflectance weight distribution map, the intensity of specular reflection received by all coordinates in an image (such as a two-dimensional image) is the same, that is, the brightness will be the same regardless of the area. However, in fact, for portraits, the specular reflectance of hair will be much lower than the specular reflectance of the highlights of the face, such as the apple muscles. Therefore, based on the real principle of light and shadow, it can be seen that different areas of a three-dimensional object have different reflectances to light. Specifically, under the same lighting conditions, the brightness of different areas in the image is different.

[0061] For example, in a portrait scene, if the reflectivity of light is the same, that is, the reflectivity of light hitting the eyeball and the nose is the same, then when the two-dimensional image is rendered with light, under the same lighting conditions, the eyes, which should be brighter, will appear slightly dim.

[0062] To make the rendered image more realistic, a distribution map of the light reflectance weights of the foreground image in the 2D image can be obtained. This allows the foreground image's light reflectance weights to be incorporated into the rendering process, allowing different areas of the foreground object in the 2D or 3D image to reflect different specular lighting information, thereby simulating more realistic lighting details. This better simulates the reflection of light from the subject (i.e., the object in the foreground image), resulting in a more realistic lighting effect for the rendered subject.

[0063] For example, assuming that the light reflectance weight of the eye area in the foreground image of a two-dimensional image is greater than the light reflectance weight of the skin area, after rendering the two-dimensional image based on the light reflectance weight distribution map of the foreground image and the specific light intensity, the brightness of the eye area in the rendered two-dimensional image can be made higher than the brightness of the skin area, thereby making the rendered two-dimensional image more realistic.

[0064] In some embodiments of the present application, the electronic device may calculate a first normal map of a foreground image in a two-dimensional image by including the following steps A and B.

[0065] Step A: The electronic device calculates a second normal map of the two-dimensional image.

[0066] In some embodiments of the present application, the electronic device may input a two-dimensional image into a normal estimation model, and perform normal inference on the two-dimensional image through the normal estimation model to obtain a second normal map of the two-dimensional image.

[0067] In some embodiments of the present application, a normal estimation model is obtained by training a neural network model using normal calculation sample data. The normal calculation sample data includes multiple training data pairs, each of which includes: an input image and a label image, where the label image is a normal map of the input image.

[0068] In some embodiments of the present application, the normal estimation model may include P convolution modules and P deconvolution modules, where P is a positive integer. For example, the normal estimation model may be the neural network model shown in FIG3 .

[0069] In step B, the electronic device uses the hair segmentation image of the foreground image in the two-dimensional image to perform cropping processing on the second normal map to obtain a first normal map.

[0070] In some embodiments of the present application, an electronic device can segment the foreground and background images of a two-dimensional image using a hair segmentation model to obtain a hair segmentation image of the foreground image in the two-dimensional image. It is understood that using a hair segmentation model can improve the segmentation accuracy of the foreground and background images of the two-dimensional image.

[0071] In some embodiments of the present application, a hair segmentation model can be obtained by training a neural network model using hair segmentation sample data. The hair segmentation sample data includes multiple training data pairs, each of which includes an input image and a label image, where the label image is a segmented hair image of the foreground image in the input image.

[0072] In some embodiments of the present application, the hair segmentation model may include N convolution modules and N deconvolution modules, where N is a positive integer. For example, the hair segmentation model may be the neural network model shown in FIG3 .

[0073] In some embodiments of the present application, the hair segmentation image may also be referred to as a mask image of a foreground image in a two-dimensional image, in which pixel values ​​of an image region corresponding to the foreground image are fixed values.

[0074] For example, in the mask image of the foreground image, the pixel value of the image area corresponding to the foreground image is 1, and the pixel value of the remaining area is 0.

[0075] It can be understood that the hair segmentation image of the foreground image in the two-dimensional image is to better focus the lighting information on the foreground area of ​​the two-dimensional image in the subsequent rendering process, and no further processing is performed on the lighting of the background area.

[0076] In some embodiments of the present application, the electronic device may crop an image area other than an image area corresponding to the foreground image in the second normal map based on a mask image of the foreground image.

[0077] In some embodiments of the present application, the above steps A and B are illustrated by taking the example of an electronic device first obtaining a second normal map of the entire two-dimensional image, and then cropping the second normal map using a mask image of the foreground image. In actual implementation, the mask image of the foreground image in the two-dimensional image can also be directly used to crop the image area other than the foreground image in the two-dimensional image to obtain the foreground image; and then the foreground image is input into the normal estimation model for processing to obtain the second normal map of the foreground image in the two-dimensional image.

[0078] In this way, since the segmentation accuracy of the hair segmentation image of the foreground image in the two-dimensional image is relatively high, the hair segmentation image of the foreground image is used to crop the normal map of the two-dimensional image, so that the cropped normal map includes the normal vectors of all details in the foreground image, thereby improving the accuracy of the normal image of the foreground image.

[0079] In some embodiments of the present application, the electronic device calculates the light reflectance weight distribution map of the foreground image in the two-dimensional image, which may include the following steps C and D.

[0080] Step C: The electronic device identifies M element images in the foreground image.

[0081] Wherein, M can be a positive integer.

[0082] It is understood that the M element images may be images of at least some elements in the foreground image. For example, the M element images may be key elements in the foreground image, such as images of hair.

[0083] In some embodiments of the present application, an object is composed of object elements.

[0084] For example, if the foreground image includes a person image, the object elements of the person may include at least one of the following: skin, hair, facial features, apple cheeks, arms, hands, legs, clothes, etc.

[0085] For example, assuming that the foreground image includes a tree image, the object elements of the tree may include at least one of the following: branches, trunks, and leaves.

[0086] For example, if the foreground image includes an image of food, the object elements of the food can be distinguished according to the type, color or shape of the ingredients. For example, when distinguished by the type of ingredients, the object elements of the food can include but are not limited to: apples, bananas, dragon fruits, grapes, lychees, eggs, green vegetables, etc.

[0087] In some embodiments of the present application, the electronic device may first identify object elements in a foreground image, and then segment the foreground image based on the object elements to obtain M element images.

[0088] In some embodiments of the present application, the electronic device may use image recognition technology to identify object elements in the foreground image; or, the electronic device may identify or determine object elements in the foreground image based on a neural network model.

[0089] For example, taking the foreground image as a person image, the electronic device can identify or determine the object element area in the foreground image through the following steps 1 to 3:

[0090] Step 1: The electronic device performs skin recognition processing on the two-dimensional image to obtain the skin area in the foreground image.

[0091] In some embodiments of the present application, the electronic device may input a two-dimensional image into a skin segmentation model, which may perform skin segmentation processing on the two-dimensional image to obtain a mask image of the skin region. The electronic device may know the location of the skin region based on the mask image.

[0092] In some embodiments of the present application, a skin segmentation model is obtained by training a neural network model using skin segmentation sample data. The skin segmentation sample data includes multiple training data pairs, each of which includes an input image and a label image, where the label image is an image of the skin region of the input image.

[0093] In some embodiments of the present application, the skin segmentation model may include K convolution modules and K deconvolution modules, where K is a positive integer. For example, the skin segmentation model may be the neural network model shown in FIG3 .

[0094] Step 2: The electronic device performs face analysis on the two-dimensional image to obtain the facial features and hair areas in the foreground image.

[0095] In some embodiments of the present application, the electronic device may input the two-dimensional image into a face parsing model, and the face parsing model may perform face parsing processing on the two-dimensional image to obtain mask images of the facial features and hair areas in the foreground image.

[0096] It can be understood that based on the mask images of the facial features area and the hair area in the foreground image, the distribution areas of the eyes, mouth and hair in the foreground image can be determined.

[0097] For the description of the face parsing model, please refer to the relevant description of the skin segmentation model in the above embodiment. To avoid repetition, it will not be repeated here.

[0098] Step 3: The electronic device performs facial key point recognition processing on the two-dimensional image to obtain a facial key point set of the character image, and determines the apple muscle area in the foreground image based on the facial key point set.

[0099] In some embodiments of the present application, an electronic device may input a two-dimensional image into a facial key point model, which may then extract feature information of facial key points in the two-dimensional image to obtain a facial key point set. The electronic device may then fit the apple cheek region in the foreground image based on the feature information in the facial key point set.

[0100] In some embodiments of the present application, the facial landmark model can extract feature information of dense facial landmarks in a two-dimensional image. For example, the facial landmark model can extract feature information of 468 facial landmarks. This allows for accurate fitting of the apple cheek region of a face based on the feature information in the facial landmark set.

[0101] The image processing method provided in the embodiments of the present application is exemplified below with reference to the accompanying drawings.

[0102] For example, if the two-dimensional image is the face image shown in Figure 4, then the electronic device can input the face image into the skin segmentation model, the face parsing model and the face key point model respectively for processing to output mask images of the skin area, facial features area and hair area in the foreground image and a set of face key points, so that the object elements in the face image can be determined based on these mask images and the set of face key points as shown in Figure 5. It can be seen from Figure 5 that the skin area 50, the eyebrow area 51, the eye area 52, the nose area 53, the mouth area 54, the hair area 55 and the apple muscle area 56 in the face image.

[0103] Step D: The electronic device sets the light reflectance weights of the M element images in a manner such that the light reflectance weights decrease from the middle area to the edge area of ​​the element image, and obtains a light reflectance weight distribution map.

[0104] The light reflectance weight of the middle area of ​​each element image is determined by the element attribute of the object element in each element image.

[0105] In some embodiments of the present application, the element attributes of the object element may include, but are not limited to, the type of the element, the color of the element, the material of the element, and other attributes that can indicate the light reflectivity of the element.

[0106] For example, taking the color of an element as an example, black has a lower reflectivity to light, while white has a higher reflectivity to light.

[0107] For another example, taking the material of an element as an example, a material with a smooth surface has a higher reflectivity to light, while a material with a bumpy surface has a lower reflectivity to light.

[0108] For another example, taking the type of element as an example, according to the reflectivity of various pixel elements in the portrait to light, it can be known that the reflectivity of the eyes, apple muscles, nose, hair, skin and lips to light decreases in turn, so the light reflectivity weight of the middle area of ​​the eyeball can be set to 0.8, the light reflectivity weight of the middle area of ​​the apple muscles can be set to 0.6, the light reflectivity weight of the middle area of ​​the hair can be set to 0.4, the light reflectivity weight of the skin area excluding the apple muscles can be set to 0.3, and the light reflectivity weight of the middle area of ​​the lips can be set to 0.1.

[0109] In some embodiments of the present application, in order to ensure that the reflection of light by each object element in the subsequently rendered image is more natural, the light reflectance weights at the boundaries of adjacent element images may be set to be similar.

[0110] In some embodiments of the present application, the line connecting the center points of two element images can be segmented using the boundary line of two adjacent images to obtain two line segments. The light reflectance weights at the edges of the two element images can then be determined based on the lengths of the two line segments and the light reflectance weights of the middle regions of the two element images. This is then combined with the light reflectance weights at the edges, the light reflectance weights of the center region, and the distance from the center region to the edge region to perform a light reflectance weight interpolation operation on each element image.

[0111] In some embodiments of the present application, assuming that the endpoint coordinates of a line segment are (x1, y1) and (x2, y2), the length of the line segment is

[0112] In some embodiments of the present application, the light reflectance weights at the boundaries of two adjacent images can be set. For example, the light reflectance weight at the edge of the eye can be set to 0.2. The light reflectance weight at the edge of the skin region can be set to 0.22. The light reflectance weights at the edge and the center region are then combined to perform a light reflectance weight interpolation operation on each element image.

[0113] It should be noted that when a two-dimensional image includes multiple identical elements and the areas where these elements are located in the two-dimensional image are not adjacent, each area including the element is regarded as an element image, for example, the left eye and the right eye are respectively regarded as an element image.

[0114] In this way, since the electronic device can set the light reflectance weights of M element images in units of element images in a manner such that the light reflectance weights decrease from the middle area to the edge area of ​​the element image, it can be ensured that in the subsequent rendering process, the brightness between the M element images is more realistic, and the brightness of different areas in the same element image is also more realistic.

[0115] Step 102: The electronic device performs lighting rendering processing on the first image based on the first normal map, the light reflectance weight distribution map, and the first light source attribute information.

[0116] The first image may include a two-dimensional image or a three-dimensional image corresponding to the two-dimensional image.

[0117] It can be understood that the electronic device can directly perform lighting rendering processing on the two-dimensional image based on the first normal map lighting reflectance weight distribution map and the first light source attribute information; or the electronic device can directly perform lighting rendering processing on the three-dimensional image corresponding to the two-dimensional image based on the first normal map lighting reflectance weight distribution map and the first light source attribute information, that is, perform three-dimensional lighting rendering processing.

[0118] Optionally, the first light source attribute information may be light source attribute information of any possible light source.

[0119] In some embodiments of the present application, the electronic device can input the first normal map, the light reflectance weight distribution map and the first image into a renderer, so that the renderer performs lighting rendering processing on the first image based on the first normal map, the light reflectance weight distribution map and the first light source attribute information.

[0120] It should be noted that, because the first image is rendered based on the first normal map, the light reflectance weight distribution map, and the first light source attribute information of the foreground image in the two-dimensional image, the lighting information can be focused on the foreground object in the first image, and the lighting information corresponding to the first light source attribute information can be more realistically rendered in detail on the foreground object. In other words, in the embodiment of the present application, the electronic device performs lighting rendering processing on the foreground object in the first image based on the first normal map, the light reflectance weight distribution map, and the first light source attribute information.

[0121] It can be understood that since the foreground image includes the main object in the first image, after the electronic device performs lighting rendering processing on the first image based on the first normal map, the light reflectance weight distribution map and the first light source attribute information, on the one hand, it can ensure that the lighting information is concentrated on the main object in the first image; on the other hand, it can make the performance of the lighting information on the main object more realistic and in line with the principles of light and shadow, thereby achieving a more realistic lighting effect.

[0122] Optionally, the first light source attribute information is determined according to at least one of the following: a first method, light source attribute information of a main light source in the first image; a second method, input parameters input by a user.

[0123] The light source attribute information may include at least one of the following: light source direction, light source position, light source intensity, and color temperature.

[0124] In some embodiments of the present application, the electronic device may perform illumination calculation on the first image to obtain light source attribute information of the main light source in the first image.

[0125] For example, taking the first image as a three-dimensional image corresponding to the two-dimensional image, the electronic device can analyze the 3D scene information of the three-dimensional image to obtain the ambient brightness and the light source direction, light source intensity and color temperature of the main light source in the three-dimensional image.

[0126] For example, the electronic device may input the first image into an illumination estimation model to calculate light source property information of a main light source in the first image through the illumination estimation model.

[0127] The following describes how the electronic device determines the first light source attribute information in different situations.

[0128] Case 1: The first light source attribute information is determined according to the light source attribute information of the main light source in the first image.

[0129] In some embodiments of the present application, the electronic device may determine the light source direction of the fill light source according to the light source direction of the main light source in the first image.

[0130] For example, the electronic device can add a fill light source diagonally to the main light source to act as a fill light panel. It is understood that the light source direction of the fill light source is symmetrical with the light source direction of the main light source. For example, the light source direction of the fill light source and the light source direction of the main light source are symmetrical with respect to the foreground object in the first image.

[0131] In some embodiments of the present application, the electronic device can enhance the light source intensity or color temperature of the main light source in the first image to enhance the original light source in the same direction as the main light source, thereby achieving the effect of enhancing lighting and three-dimensional perception.

[0132] For example, the electronic device may increase the light source intensity of the main light source by 2 times.

[0133] For example, the electronic device may increase the color temperature of the primary light source by a factor of 1.5.

[0134] Case 2: The light source attribute information is determined based on the input parameters entered by the user.

[0135] In some embodiments of the present application, before performing lighting rendering processing on the first image, the electronic device may first display the first image for user preview, such as mapping the three-dimensional first image (i.e., the first image is a three-dimensional image corresponding to the two-dimensional image) to a two-dimensional mapping image. The user may then trigger the electronic device to determine the first light source attribute information through input.

[0136] Example 1, as shown in FIG6(A), the user may trigger the electronic device to add a white fill light at position 61 in the three-dimensional image 60.

[0137] Example 2, as shown in FIG6(B), the user may trigger the electronic device to add an orange fill light at position 62 in the three-dimensional image 60.

[0138] Example 3, as shown in FIG6(C), the user can trigger the electronic device to add an orange backlight at position 63 in the three-dimensional image 60, that is, position 63 is located at the back of the portrait 64.

[0139] Example 4, as shown in FIG6(D), the user can trigger the electronic device to add an orange fill light at position 65 in the three-dimensional image 60 and to add a blue fill light at position 66 in the three-dimensional image 60.

[0140] The above examples 1 to 4 are illustrated by taking the first image including a portrait as an example. For the description of 3D lighting for landscape and food images, please refer to the relevant description of 3D lighting for portrait images.

[0141] In some embodiments of the present application, an electronic device may display a first image and at least one control, each of which may include a color temperature control, a light intensity control, a light source position control, or a light source direction control. Thus, a user may trigger the electronic device to determine first light source attribute information by inputting a target control among the at least one control, and perform lighting rendering processing on the first image based on the first normal map, the light reflectance weight distribution map, and the first light source attribute information.

[0142] For example, the electronic device can adjust information such as color temperature, light intensity, depth (position of a newly added light source), and light source direction based on user input to a target control.

[0143] In some embodiments of the present application, when the at least one control includes a light source direction control, the user may first input a direction in the light source direction control and then swipe on the first image to trigger the electronic device to use the direction of the user's swipe input as the light source direction of the fill light. Of course, the light source direction control may not be provided. In this case, swiping on the first image indicates that the user wishes to add a fill light in the direction of the swipe input.

[0144] In some embodiments of the present application, when the at least one control includes a light source direction control, the light source direction control may include multiple sub-controls, each sub-control being used to indicate a fill light direction or a fill light effect.

[0145] Exemplarily, the light source direction control may include: a backlight control, a fill light control, and a same-direction fill light control.

[0146] The backlight control is used to trigger the electronic device to set a fill light between the foreground object and the background object in the three-dimensional image, so as to produce a contour light effect through the fill light;

[0147] The fill light control can be used to trigger the electronic device to set a fill light on the front or side of the foreground object to increase the brightness of the foreground object;

[0148] The same-direction fill light control can be used to trigger the electronic device to add a fill light in the direction of the main light source in the first image to enhance the original light source, thereby achieving the effect of enhancing lighting and three-dimensional effect.

[0149] In some embodiments of the present application, the light source intensity control may include: a first sliding area and a first slider set on the first sliding area, wherein the first sliding area indicates the light intensity adjustment range, and the position of the first slider on the sliding area indicates the light intensity selected by the user.

[0150] In some embodiments of the present application, the color temperature control may include: a second sliding area and a second slider set on the second sliding area, wherein the second sliding area indicates the color temperature adjustment range, and the position of the first slider on the second sliding area indicates the color temperature selected by the user.

[0151] In some embodiments of the present application, the light source position control may include: a third sliding area and a third slider set on the third sliding area, wherein the third sliding area indicates the light source position adjustment range, and the position of the third slider on the third sliding area indicates the light source position selected by the user.

[0152] Case 3: The first light source attribute information is determined according to the light source attribute information of the main light source in the first image and input parameters input by the user.

[0153] In some embodiments of the present application, before performing lighting rendering on the first image, the electronic device may display the first image for user preview. Furthermore, after obtaining light source attribute information of the primary light source in the first image, the electronic device may determine, based on this light source attribute information, a default parameter value or recommended parameter range corresponding to at least one control in scenario 2. The user may then trigger the electronic device to determine the default parameter value as the first light source attribute information or to redefine the light source attribute information by inputting into the at least one control.

[0154] For example, the illumination intensity corresponding to the first sliding area may be set to [0*n, 1.5*n], wherein the recommended illumination intensity is at 0.8*n, and n is the light source intensity of the main light source in the first image.

[0155] For example, the recommended light source position of the third sliding area may be set to (5.0*x, 4*y, z), where (x, y, z) is the light source position of the main light source in the first image.

[0156] Step 103: The electronic device displays the rendered first image.

[0157] Optionally, the electronic device displaying the rendered first image may include: the electronic device displaying the rendered two-dimensional image; or the electronic device displaying a two-dimensional mapping image corresponding to the rendered three-dimensional image. For example, after the electronic device renders the three-dimensional image, it may map the rendered three-dimensional image, such as mapping it to the display screen of the electronic device to obtain a two-dimensional mapping image.

[0158] In the image processing method provided in the embodiment of the present application, a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image can be calculated; based on the first normal map, the light reflectance weight distribution map, and the first light source attribute information, light rendering processing is performed on the first image, the first image including the two-dimensional image or the three-dimensional image corresponding to the two-dimensional image; and the rendered first image is displayed. Through this solution, when it is required to perform light rendering on a two-dimensional image, since the first normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image are first calculated, and then based on the normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image, the two-dimensional image or the three-dimensional image corresponding to the two-dimensional image is light rendered, it can be ensured that the rendered image has a good three-dimensional sense, thereby improving the display effect of the image after adding the light effect.

[0159] In some embodiments of the present application, when the first image is a three-dimensional image corresponding to a two-dimensional image, this indicates that the electronic device adds 3D lighting effects to the existing 2D image, achieving a 360-degree realistic lighting effect (supporting the addition of backlights behind the subject / portrait to create a contour light effect). Furthermore, it supports the adjustment of color temperature and color, as well as the addition of multiple light sources, enabling more possibilities for ambient lighting. Furthermore, it expands user editing capabilities, allowing for simple and convenient addition of three-dimensional lighting, turning discarded images into valuable content and transforming ordinary images into blockbusters with just one click. This has strong practical value and effectively enhances the user experience.

[0160] Optionally, the first image may include a three-dimensional image corresponding to the two-dimensional image; in combination with the above-mentioned Figure 1, as shown in Figure 7, before the above-mentioned step 102, the image processing method provided in the embodiment of the present application may further include the following step 104.

[0161] Step 104 : The electronic device performs three-dimensional reconstruction processing on the two-dimensional image according to the depth estimation information of the two-dimensional image to obtain a three-dimensional image corresponding to the two-dimensional image.

[0162] It is understood that the depth estimation information of a 2D image can provide the depth relationship between objects in the 2D image and obtain the positional relationship between the objects. Therefore, based on the depth estimation information of the 2D image, the 2D image can be reconstructed into 3D to obtain a 3D image corresponding to the 2D image.

[0163] It should be noted that the above three-dimensional image can also be called a three-dimensional scene corresponding to the two-dimensional image.

[0164] It should be noted that, since the depth estimation information of the 2D image is obtained based on the 2D image, the depth estimation information of the 2D image includes partial depth information of visible objects in the 2D image. For example, assuming that the 2D image includes the face of person 1, the depth estimation information of the 2D image includes the depth estimation information of person 1's face, but does not include the depth estimation information of the back of person 1's head. Therefore, the above-mentioned 3D image includes the 3D object corresponding to the visible object in the 2D image, such as the 3D face.

[0165] In some embodiments of the present application, the spatial relationship of objects in a three-dimensional image may be adjusted by adjusting depth estimation information of a two-dimensional image.

[0166] In some embodiments of the present application, after obtaining a three-dimensional image corresponding to a two-dimensional image, a stereoscopic editable file of the three-dimensional image, such as an obj file of the three-dimensional image, may be exported.

[0167] In some embodiments of the present application, depth estimation information of a two-dimensional image is obtained by calculating a depth estimation model. The specific method for calculating the two-dimensional image based on the depth estimation model will be described in detail in the following embodiments.

[0168] FIG7 illustrates an example of executing step 104 after executing step 101. In actual implementation, step 104 may be executed first and then step 101; or step 101 and step 104 may be executed simultaneously, depending on specific usage requirements.

[0169] In this way, when it is necessary to perform lighting rendering on a two-dimensional image, the three-dimensional image corresponding to the two-dimensional image can be determined first based on the estimated depth information of the two-dimensional image, and then the three-dimensional lighting rendering is performed on the three-dimensional image based on the normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image. Therefore, it can be ensured that the two-dimensional mapping image of the rendered three-dimensional image has a three-dimensional lighting effect rendering effect, that is, the addition of three-dimensional lighting effects to the two-dimensional image is achieved, which can improve the display effect of the image after adding the lighting effect.

[0170] In some embodiments of the present application, before the above step 104, the image processing method provided by the embodiment of the present application may further include the following step 105.

[0171] Step 105 : The electronic device inputs the two-dimensional image into a depth estimation model, calculates depth information of the two-dimensional image through multiple feature extraction modules in the depth estimation model in sequence, and outputs depth estimation information of the two-dimensional image.

[0172] Among them, each feature extraction module in the depth estimation model may include a convolutional layer and a visual transformation Vit module connected to each other.

[0173] In some embodiments of the present application, the depth calculation module may further include multiple deconvolution modules, and the number of the deconvolution modules is the same as the number of the convolution modules.

[0174] It is understood that depth calculation in related technologies generally requires hardware (such as a binocular camera) to measure the depth of an object from the camera. In the image processing method provided in this application, the result of the depth calculation is predicted by a deep learning network.

[0175] The depth estimation model uses a UNET network as the base network model, which includes U convolutional layers and U deconvolution modules. A VIT module (transformer network structure) is connected in series after each convolutional layer. For example, the depth estimation model can include 3 feature extraction modules and 3 deconvolution modules.

[0176] It can be understood that the VIT module contains a multi-head attention structure. The multi-head attention structure can improve the expressive power of the attention mechanism and has an important contribution to the performance of the transformer model. It can effectively improve the model's ability to express and understand sequence information.

[0177] Furthermore, the present application performs operations such as distillation and pruning on the depth estimation model to compress the depth estimation model, thereby obtaining a lightweight model to support the depth estimation model running on a mobile terminal.

[0178] In some embodiments of this application, the depth calculation surface model inference time is less than 200ms, effectively improving the network's prediction accuracy while ensuring performance. After obtaining the trained depth estimation model, the original 2D image can be inferred to obtain the depth calculation result.

[0179] In this way, since the electronic device can calculate the depth estimation information of the two-dimensional image through a depth calculation module including multiple feature extraction modules, compared with the solution of estimating the depth information through multiple 2D images or uploading the depth estimation information to the server, the image processing method provided in the embodiment of the present application can complete the calculation of the depth information on the terminal side, thereby improving the operational convenience of the depth calculation.

[0180] In some embodiments of the present application, the electronic device can use a neural network model, also known as an artificial intelligence (AI) lightweight model, to calculate depth and normal surface information on a 2D image, and perform three-dimensional reconstruction of the entire scene of the 2D image, which includes depth calculation, model reconstruction process, and obtaining an obj file; and supports previewing its 3D scene information, and can use the superposition of surface normal information to reflect the real light reflection information; further optionally, it can support regional calculation for the portrait area to simulate the real light reflectivity, so that the later virtual lighting can be more realistic. After that, we analyze the obtained 3D scene information, analyze the ambient brightness and the direction and intensity information of the original image's main light, and then combine the structure of the original image to automatically place the light effect. At the same time, it supports users to add light sources and secondary edit the light source properties of the fill light (light position, direction, color temperature, and color light).

[0181] The image processing methods provided in the embodiments of this application include the following: 1) Existing methods for adding 3D lighting effects require designers (modelers) to manually reconstruct the 3D scene, or to use neural rendering to reconstruct the 3D scene using multiple RGB images (continuous surround shooting). The former is time-consuming and labor-intensive, and the latter suffers from severe loss of resolution and other precision, making it unfeasible for mobile applications. However, in this application, for any 2D image, there is no need to upload a detailed 3D model or perform cloud-based processing and calculations, making it very convenient for users to operate and reduce waiting time. 2) The application adds parameters such as the normal surface and reflectivity of the foreground image to more realistically simulate the lighting effects of a real fill light. 3) Multiple fill light sources and ambient lighting can be set to enhance the user experience and facilitate user operation. 4) In addition to analyzing the light source properties of the main light source based on the original image and enhancing the original light source in the same direction as the main light to achieve enhanced lighting and three-dimensional effects, fill light sources can also be added diagonally to the main light to act as fill light panels. 5) The color and color temperature can be adjusted to achieve an atmospheric photo effect.

[0182] It can be understood that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0183] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0184] An embodiment of the present application provides an image processing device. FIG8 shows a schematic structural diagram of the image processing device provided by the embodiment of the present application. As shown in FIG8 , the image processing device 80 may include: a processing module 81 , a rendering module 82 and a display module 83 .

[0185] A processing module 81 is used to calculate a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image;

[0186] a rendering module 82 configured to perform illumination rendering on the first image processed by the processing module 81 based on the first normal map, the illumination reflectance weight distribution map, and the first light source attribute information, wherein the first image includes the two-dimensional image or a three-dimensional image corresponding to the two-dimensional image;

[0187] The display module 83 is configured to display the first image rendered by the rendering module 82 .

[0188] In the image processing device provided in the embodiment of the present application, when it is required to perform lighting rendering on a two-dimensional image, the first normal map and the lighting reflectance weight distribution map of the foreground image in the two-dimensional image are first calculated, and then the two-dimensional image or the three-dimensional image corresponding to the two-dimensional image is light rendered based on the normal map and the lighting reflectance weight distribution map of the foreground image in the two-dimensional image. Therefore, it can be ensured that the rendered image has a better three-dimensional sense, thereby improving the display effect of the image after adding lighting effects.

[0189] In some embodiments of the present application, the processing module is specifically configured to calculate a second normal map of the two-dimensional image; and to crop the second normal map using a hair segmentation image of the foreground image to obtain the first normal map.

[0190] In some embodiments of the present application, the processing module is specifically configured to identify M element images in the foreground image, where M is a positive integer;

[0191] Setting the light reflectance weights of the M element images in a manner such that the light reflectance weights decrease from the middle area to the edge area of ​​the element image, to obtain the light reflectance weight distribution map;

[0192] The light reflectance weight of the middle area of ​​each element image is determined by the element attribute of the object element in each element image.

[0193] In some embodiments of the present application, the first light source attribute information is determined according to at least one of the following: light source attribute information of a main light source in the first image, an input parameter input by a user;

[0194] The light source attribute information includes at least one of the following: light source direction, light source position, light source intensity, and color temperature.

[0195] In some embodiments of the present application, the first image includes a three-dimensional image corresponding to the two-dimensional image;

[0196] The processing module is also used to perform three-dimensional reconstruction processing on the two-dimensional image according to the depth estimation information of the two-dimensional image before the rendering module performs lighting rendering processing on the first image based on the first normal map, the lighting reflectance weight distribution map and the first light source attribute information, so as to obtain a three-dimensional image corresponding to the two-dimensional image.

[0197] In some embodiments of the present application, the processing module is further configured to, before performing three-dimensional reconstruction processing on the two-dimensional image based on the depth estimation information of the two-dimensional image to obtain a three-dimensional image corresponding to the two-dimensional image, input the two-dimensional image into a depth estimation model, sequentially calculate the depth information of the two-dimensional image using multiple feature extraction modules in the depth estimation model, and output the depth estimation information of the two-dimensional image;

[0198] Among them, each feature extraction module includes a convolution layer and a Vit module connected together.

[0199] The image processing device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0200] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0201] The image processing device provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 7. To avoid repetition, they will not be described here.

[0202] In some embodiments of the present application, as shown in Figure 9, the embodiment of the present application also provides an electronic device 900, including a processor 901 and a memory 902, and the memory 902 stores a program or instruction that can be run on the processor 901. When the program or instruction is executed by the processor 901, the various steps of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0203] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0204] FIG10 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0205] The electronic device 1500 includes but is not limited to components such as a radio frequency unit 1501 , a network module 1502 , an audio output unit 1503 , an input unit 1504 , a sensor 1505 , a display unit 1506 , a user input unit 1507 , an interface unit 1508 , a memory 1509 , and a processor 1510 .

[0206] Those skilled in the art will appreciate that the electronic device 1500 may further include a power source (e.g., a battery) to power various components. The power source may be logically connected to the processor 1510 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG10 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or arrange the components differently, which will not be described in detail here.

[0207] The processor 1510 is configured to calculate a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image;

[0208] Processor 1510, configured to perform lighting rendering processing on a processed first image based on the first normal map, the lighting reflectance weight distribution map, and the first light source attribute information, where the first image includes the two-dimensional image or a three-dimensional image corresponding to the two-dimensional image;

[0209] The display unit 1506 is configured to display the first image rendered by the processor 1510 .

[0210] In the image processing device provided in the embodiment of the present application, when it is required to perform lighting rendering on a two-dimensional image, the first normal map and the lighting reflectance weight distribution map of the foreground image in the two-dimensional image are first calculated, and then the two-dimensional image or the three-dimensional image corresponding to the two-dimensional image is light rendered based on the normal map and the lighting reflectance weight distribution map of the foreground image in the two-dimensional image. Therefore, it can be ensured that the rendered image has a better three-dimensional sense, thereby improving the display effect of the image after adding lighting effects.

[0211] In some embodiments of the present application, the processor 1510 is specifically configured to calculate a second normal map of the two-dimensional image; and use the hair segmentation image of the foreground image to crop the second normal map to obtain the first normal map.

[0212] In some embodiments of the present application, the processor 1510 is specifically configured to identify M element images in the foreground image, where M is a positive integer;

[0213] Setting the light reflectance weights of the M element images in a manner such that the light reflectance weights decrease from the middle area to the edge area of ​​the element image, to obtain the light reflectance weight distribution map;

[0214] The light reflectance weight of the middle area of ​​each element image is determined by the element attribute of the object element in each element image.

[0215] In some embodiments of the present application, the first light source attribute information is determined according to at least one of the following: light source attribute information of a main light source in the first image, an input parameter input by a user;

[0216] The light source attribute information includes at least one of the following: light source direction, light source position, light source intensity, and color temperature.

[0217] In some embodiments of the present application, the first image includes a three-dimensional image corresponding to the two-dimensional image;

[0218] The processor 1510 is also used to perform three-dimensional reconstruction processing on the two-dimensional image according to the depth estimation information of the two-dimensional image before performing lighting rendering processing on the first image based on the first normal map, the lighting reflectance weight distribution map and the first light source attribute information to obtain a three-dimensional image corresponding to the two-dimensional image.

[0219] In some embodiments of the present application, the processor 1510 is further configured to, before performing 3D reconstruction on the 2D image based on the depth estimation information of the 2D image to obtain a 3D image corresponding to the 2D image, input the 2D image into a depth estimation model, sequentially calculate the depth information of the 2D image using multiple feature extraction modules in the depth estimation model, and output the depth estimation information of the 2D image;

[0220] Among them, each feature extraction module includes a convolution layer and a Vit module connected together.

[0221] It should be understood that in an embodiment of the present application, the input unit 1504 may include a graphics processing unit (GPU) 15041 and a microphone 15042, and the graphics processor 15041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1506 may include a display panel 15061, and the display panel 15061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include two parts: a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0222] The memory 1509 can be used to store software programs and various data. The memory 1509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1509 may include a volatile memory or a non-volatile memory, or the memory 1509 may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0223] Processor 1510 may include one or more processing units. Optionally, processor 1510 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1510.

[0224] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0225] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0226] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0227] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0228] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0229] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0230] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0231] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, the method comprising: Calculate the first normal map and the light reflectance weight distribution map of the foreground image in the two-dimensional image; Based on the first normal map, the light reflectance weight distribution map and the first light source attribute information, performing light rendering processing on a first image, the first image including the two-dimensional image or a three-dimensional image corresponding to the two-dimensional image; The rendered first image is displayed.

2. The method according to claim 1, wherein: Calculating a first normal map of a foreground image in the two-dimensional image, comprising: Calculating a second normal map of the two-dimensional image; The second normal map is cropped using the hair segmentation image of the foreground image to obtain the first normal map.

3. The method according to claim 1, wherein: Calculating a light reflectance weight distribution map of a foreground image in the two-dimensional image, comprising: Identify M element images in the foreground image, where M is a positive integer; The light reflectance weights of the M element images are set in a manner such that the light reflectance weights decrease from the middle area to the edge area of ​​the element image, so as to obtain the light reflectance weight distribution map; The light reflectance weight of the middle area of ​​each element image is determined by the element attribute of the object element in each element image.

4. The method according to claim 1, wherein: The first light source attribute information is determined according to at least one of the following: light source attribute information of a main light source in the first image, an input parameter input by a user; The light source attribute information includes at least one of the following: light source direction, light source position, light source intensity, and color temperature.

5. The method according to any one of claims 1 to 4, wherein: The first image includes a three-dimensional image corresponding to the two-dimensional image; Before performing illumination rendering processing on the first image based on the first normal map, the illumination reflectance weight distribution map and the first light source attribute information, the method further includes: According to the depth estimation information of the two-dimensional image, three-dimensional reconstruction processing is performed on the two-dimensional image to obtain a three-dimensional image corresponding to the two-dimensional image.

6. The method according to claim 5, wherein: Before performing three-dimensional reconstruction processing on the two-dimensional image according to the depth estimation information of the two-dimensional image to obtain a three-dimensional image corresponding to the two-dimensional image, the method further includes: Inputting the two-dimensional image into a depth estimation model, calculating depth information of the two-dimensional image through a plurality of feature extraction modules in the depth estimation model in sequence, and outputting depth estimation information of the two-dimensional image; Among them, each feature extraction module includes a convolution layer and a visual transformation Vit module connected to each other.

7. An image processing device, comprising: Processing module, rendering module and display module; The processing module is used to calculate a first normal map and a light reflectance weight distribution map of a foreground image in a two-dimensional image; The rendering module is used to perform lighting rendering processing on the first image processed by the processing module based on the first normal map, the lighting reflectance weight distribution map and the first light source attribute information, wherein the first image includes the two-dimensional image or a three-dimensional image corresponding to the two-dimensional image; The display module is used to display the first image rendered by the rendering module.

8. The device according to claim 7, wherein: The processing module is specifically used to calculate the second normal map of the two-dimensional image; and use the hair segmentation image of the foreground image to crop the second normal map to obtain the first normal map.

9. The device according to claim 7, wherein: The processing module is specifically used to identify M element images in the foreground image, where M is a positive integer; The light reflectance weights of the M element images are set in a manner such that the light reflectance weights decrease from the middle area to the edge area of ​​the element image, so as to obtain the light reflectance weight distribution map; The light reflectance weight of the middle area of ​​each element image is determined by the element attribute of the object element in each element image.

10. The device according to claim 7, wherein: The first light source attribute information is determined according to at least one of the following: light source attribute information of a main light source in the first image, an input parameter input by a user; The light source attribute information includes at least one of the following: light source direction, light source position, light source intensity, and color temperature.

11. The device according to any one of claims 7 to 10, wherein: The first image includes a three-dimensional image corresponding to the two-dimensional image; The processing module is also used to perform three-dimensional reconstruction processing on the two-dimensional image according to the depth estimation information of the two-dimensional image before the rendering module performs lighting rendering processing on the first image based on the first normal map, the lighting reflectance weight distribution map and the first light source attribute information, so as to obtain a three-dimensional image corresponding to the two-dimensional image.

12. The device according to claim 11, wherein The processing module is further configured to perform three-dimensional reconstruction processing on the two-dimensional image according to the depth estimation information of the two-dimensional image, before obtaining a three-dimensional image corresponding to the two-dimensional image, input the two-dimensional image into a depth estimation model, calculate the depth information of the two-dimensional image through a plurality of feature extraction modules in the depth estimation model in sequence, and output the depth estimation information of the two-dimensional image; Among them, each feature extraction module includes a convolution layer and a visual transformation Vit module connected to each other.

13. An electronic device comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

14. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 6.

15. A chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image processing method as described in any one of claims 1 to 6.

16. A computer program product, wherein the computer program product is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 6.

Images