Image Processing Method, Apparatus, Electronic Device, and Readable Storage Medium

Abstract

The present application discloses an image processing method, apparatus, electronic device, and readable storage medium, belonging to the field of information processing. Among them, the method includes: analyzing the acquired first image to determine the depth information of the first image, the first ambient light source information, and the position information of the object in the first image; performing light elimination processing on the first three-dimensional scene map according to the first ambient light source information to obtain a second three-dimensional scene map, where the first three-dimensional scene map is constructed based on the depth information; determining light source parameter information according to the depth information, the first ambient light source information, and the position information; performing simulated light processing on the second three-dimensional scene map according to the light source parameter information to obtain a third three-dimensional scene map; generating a target image or a target video based on the third three-dimensional scene map.

Description

Technical Field

[0001] This application belongs to the field of image processing, and particularly relates to an image processing method, apparatus, electronic device, and readable storage medium. Background Art

[0002] With the development of image processing technology, more and more users use electronic devices to take pictures and create works based on the captured images. Recently, a method of reconstructing a three-dimensional image from a captured two-dimensional image has become very popular. Among them, the technology of lighting the reconstructed three-dimensional scene information to obtain a three-dimensional image generally uses a rendering engine, which requires lighting debugging of virtual three-dimensional scene information. Such a processing method requires users to adjust the light-related information by themselves. For ordinary users, this method is very cumbersome and difficult. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium, which can solve the problem that in the prior art, it is very cumbersome and difficult to adjust the light in the three-dimensional scene corresponding to the image.

[0004] In a first aspect, the embodiments of this application provide an image processing method, which includes:

[0005] Analyze the acquired first image to determine the depth information, first environmental light source information, and position information of the object in the first image;

[0006] According to the first environmental light source information, perform light elimination processing on the first three-dimensional scene map to obtain a second three-dimensional scene map, where the first three-dimensional scene map is constructed based on the depth information;

[0007] Determine the light source parameter information according to the depth information, first environmental light source information, and position information;

[0008] Perform simulated lighting processing on the second three-dimensional scene map according to the light source parameter information to obtain a third three-dimensional scene map;

[0009] Generate a target image or target video based on the third three-dimensional scene map.

[0010] In a second aspect, the embodiments of this application provide an image processing apparatus, which includes:

[0011] An analysis module, configured to analyze the acquired first image to determine the depth information, first environmental light source information, and position information of the object in the first image;

[0012] An elimination module, configured to perform illumination elimination processing on a first three-dimensional scene graph according to first environmental light source information, to obtain a second three-dimensional scene graph, where the first three-dimensional scene graph is constructed based on depth information;

[0013] A determination module, configured to determine light source parameter information according to depth information, first environmental light source information, and position information;

[0014] A simulated illumination module, configured to perform simulated illumination processing on the second three-dimensional scene graph according to the light source parameter information, to obtain a third three-dimensional scene graph;

[0015] A generation module, configured to generate a target image or a target video based on the third three-dimensional scene graph.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the method described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium, and the program product is executed by at least one processor to implement the method described in the first aspect.

[0020] In an embodiment of the present application, by analyzing the acquired first image, the depth information of the first image, the first environmental light source information, and the position information of the object in the first image are determined; according to the first environmental light source information, the first three-dimensional scene map constructed based on the depth information is subjected to light elimination processing to obtain a second three-dimensional scene map, which can automatically eliminate the light information inherent in the first image, and adaptively reconstruct the light source parameter information according to the depth information, the first environmental light source information, and the position information. Here, since the light source parameter information is determined according to the position information of the object and the first environmental light source information, the light information inherent in the first image can be considered, and the object in the first image can be better illuminated, ensuring the authenticity of the reconstructed light source parameter information. By performing simulated lighting processing on the second three-dimensional scene map according to the light source parameter information, a third three-dimensional scene map that is real, natural, and has a high degree of adaptability to illuminate the object can be obtained. Finally, based on the third three-dimensional scene map, a target image or a target video that is real, natural, and has a high degree of adaptability to illuminate the object can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a three-dimensional scene provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0024] Figure 4 is a structural diagram of an image processing device provided by an embodiment of the present application;

[0025] Figure 5 is one of the schematic diagrams of the hardware structure of an electronic device provided by an embodiment of the present application;

[0026] Figure 6 is the second schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solutions of the embodiments of the present application will be clearly described below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application fall within the scope of protection of the present application.

[0028] The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally represents an "or" relationship between the associated objects before and after.

[0029] The image processing method provided by the embodiments of this application can be applied to at least the following application scenarios, which will be described below.

[0030] With the development of image processing technology, more and more users use electronic devices to take pictures and often create works with the photos. Currently, as Figure 1 shown, three-dimensional scene information can be reconstructed based on a planar image. For the lighting method of the three-dimensional scene information reconstructed from the first image, generally, a rendering engine is used to place light sources with different brightnesses and materials in the virtual world, and then the three-dimensional scene information is lit. Such a processing method requires the user to adjust information such as the position of the light source, the brightness of the light source, and the material of the light source by themselves. For ordinary users, this method is very cumbersome and difficult.

[0031] In view of the problems in the related art, the embodiments of this application provide an image processing method, apparatus, electronic device, and storage medium, which can solve the problem that it is very cumbersome and difficult to adjust the light in the three-dimensional scene corresponding to the image in the related art.

[0032] The following will describe in detail the image processing method provided by the embodiments of this application with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0033] Figure 2 It is a flowchart of an image processing method provided by the embodiments of this application.

[0034] As Figure 2 shown, the image processing method may include step 210-step 250. This method is applied to an image processing apparatus and is specifically as follows:

[0035] Step 210, analyze the obtained first image to determine the depth information, the first environmental light source information, and the position information of the object in the first image.

[0036] Among them, specifically, a deep learning network can be used to analyze the obtained first image to determine the depth information, the first environmental light source information, and the position information of the object in the first image.

[0037] Among them, the depth information is used to represent the distance of each object in the image from the camera. The first environmental light source information is the environmental light source information when the first image is captured, that is, the original environmental light source information of the first image. The position information of the object in the first image can specifically be the coordinates of the object in the first image.

[0038] Among them, before step 210, it may further include:

[0039] Receiving a selection input of the first image by the user from the album; in response to the selection input, obtaining the first image.

[0040] In a possible embodiment, step 210 may specifically include the following steps:

[0041] Analyzing the first image to determine the depth information;

[0042] Extracting the position information from the first image according to the depth information;

[0043] Determining the first environmental light source information according to the position information.

[0044] In the step of analyzing the first image, it may specifically include the following steps: inputting the first image into a deep learning network to obtain the depth information.

[0045] Among them, the deep learning network may specifically include a high-precision depth estimation module, a subject discrimination module, a background completion module, and a dynamic perspective rendering module. The deep learning network may specifically output: a background completion map, the position information of the object, the depth information, and the position information of the camera. Among them, the background completion map refers to the image obtained after removing the object from the first image.

[0046] Among them, according to the depth information, when extracting the position information from the first image, the object closest to the camera among the objects in the first image may be determined as the object involved above.

[0047] Among them, in the above-mentioned step of extracting the position information of the object from the first image according to the depth information, it may specifically include the following steps:

[0048] Extracting the first object image from the first image according to the depth information;

[0049] Performing binarization processing on the first object image to obtain a second object image;

[0050] Performing filtering processing on the first object image to obtain a third object image;

[0051] Merging the second object image and the third object image to obtain a fourth object image;

[0052] Extract the position information of the object from the fourth object image.

[0053] To obtain more accurate position information of the object, it is necessary to process and analyze the obtained depth information. First, according to the depth information, extract the first object image from the first image. The object can be the main body closest to the camera.

[0054] Then, perform binarization processing on the first object image to obtain the second object image. Among them, the image area corresponding to the object in the second object image can be a white pixel value, and the background area can be a black pixel value. The background area can be the image area in the first object image except for the image area corresponding to the object.

[0055] At the same time, perform multiple filtering processes on the first object image to obtain a smoother edge. At this moment, extract its edge lines, and determine the image including the edge lines as the third object image.

[0056] Finally, merge the second object image and the third object image, which is equivalent to merging the image area corresponding to the object and the edge lines corresponding to the object, to obtain the fourth object image. The fourth object image includes the complete position information of the object, so the position information of the object can be extracted from the fourth object image.

[0057] Among them, analyzing the obtained first image to determine the depth information, the first environmental light source information, and the position information of the object in the first image can specifically include the following steps:

[0058] Analyze the obtained first image to determine the depth information and the position information of the object in the first image;

[0059] Construct the first three-dimensional scene information based on the depth information;

[0060] Determine the first environmental light source information according to the first three-dimensional scene information.

[0061] Among them, the first three-dimensional scene information can specifically be an obj file, and the obj file is a three-dimensional model file format.

[0062] Among them, in the above steps of determining the first environmental light source information according to the position information, it can specifically include the following steps:

[0063] According to the position information of the object, determine the brightness information of each surface of the object from the first image;

[0064] Based on the law of reflection of light, calculate the brightness information to obtain the first environmental light source information.

[0065] Combined with the determined position information of the object, the pixel values of each surface of the object can be determined from the first three-dimensional scene information, and then the brightness information of each surface of the object can be determined according to the pixel values of each surface of the object.

[0066] Then, based on the law of reflection of light, the brightness information is calculated to obtain the first ambient light source information.

[0067] When the original light source corresponding to the first ambient light source information shines on the object, reflection will occur, and the brightness information is the manifestation of the reflected light. Therefore, the incident light, that is, the first ambient light source information, can be deduced based on the law of reflection of light and the reflected light.

[0068] Among them, reflection is an optical phenomenon. It refers to the phenomenon that when light propagates to different substances, it changes its propagation direction at the interface and returns to the original substance. Light will be reflected when it encounters the surface of water, glass, and many other objects. When light changes its propagation direction at the interface between two substances and returns to the original substance, it is called the reflection of light.

[0069] The law of reflection of light includes: the angle of reflection is equal to the angle of incidence, and the angle between the incident light and the plane is equal to the angle between the reflected light and the plane. The reflected light and the incident light are on both sides of the normal. The reflected light, the incident light, and the normal are all in the same plane.

[0070] Here, according to the position information of the object, the brightness information of each surface of the object is determined from the first image. Based on the law of reflection of light, the brightness information is calculated, and the first ambient light source information can be obtained quickly and accurately by combining the information in the first image and the natural law.

[0071] In addition, in the above steps of calculating the brightness information based on the law of reflection of light to obtain the first ambient light source information, it may further include:

[0072] Convert the first image into a grayscale image to obtain the grayscale values of its grayscale image and the RGB values of the first image. The intensity information of the ambient light source is fitted through the RGB brightness values and grayscale values of the whole image. Among them, the intensity information of the ambient light source can specifically be the channel information of RGB.

[0073] Among them, the RGB color model is a color standard in the industrial field. It obtains various colors through the changes of the three color channels of red (R), green (G), and blue (B) and their superposition with each other. RGB represents red, green, and blue, which are the colors of these three channels. The proportion of the three channels in each pixel point sums up to 100%.

[0074] The channel information of RGB can be an expression of the material of a three-dimensional object. For example, the reflection effects of fur and glass on light are different, so the calculation ratio of the channel information of RGB can be provided to express the effect of the original light source hitting the object.

[0075] The channel information of RGB corresponding to each pixel can be determined by providing the gray value of the grayscale image of the first image and the RGB value of the first image.

[0076] Step 220, according to the first environmental light source information, perform light elimination processing on the first three-dimensional scene map to obtain a second three-dimensional scene map, and the first three-dimensional scene map is constructed based on depth information.

[0077] Among them, the first environmental light source information may include: the intensity of the original light source and the irradiation direction of the original light source. In this step, the light information in the first three-dimensional scene information is eliminated, and the entire three-dimensional scene is placed in a scenario without lighting to obtain the second three-dimensional scene information.

[0078] In a possible embodiment, step 220 includes:

[0079] Generate light information according to the first environmental light source information;

[0080] Eliminate the light information from the first three-dimensional scene map to obtain a second three-dimensional scene map.

[0081] Generate light information according to the first environmental light source information. Among them, the light information involved above can specifically be pixel values used to represent light. Eliminate the light pixel values from the first three-dimensional scene map to obtain a second three-dimensional scene map.

[0082] Among them, as Figure 3 shown, this step is to remove the light information brought by the first environmental light source information in the first image to obtain the second three-dimensional scene information.

[0083] Step 230, determine the light source parameter information according to the depth information, the first environmental light source information and the position information. In a possible embodiment, the light source parameter information includes the light source position and the second environmental light source information. Step 230 includes:

[0084] Determine the light source position according to the position information;

[0085] According to the depth information, determine the position information of the target area closest to the original camera from the position information of the object, and the original camera is the camera that captured the first image;

[0086] Determine the second environmental light source information according to the position information of the target area and the first environmental light source information.

[0087] Determine the light source position based on the position information, which may specifically include: determining the position information of the original camera based on the position information of the object; determining the light source position through the position information of the original camera and the position information of the object. The specific calculation method is as follows: The position information of the original camera is calculated through the intersection of the distance lines between the object and the background. The light source position is calculated by reflecting the position information of the original camera and the position information of the object.

[0088] Determine the position information of the target area closest to the original camera from the position information of the object according to the depth information. The target area may be a key part of the object.

[0089] For example, if the object is a person and the hand area of the person is the closest to the original camera, the hand area can be determined as the target area, and the position information of the target area can be determined from the position information of the object.

[0090] Determine the second ambient light source information according to the position information of the target area and the first ambient light source information. Specifically, the first ambient light source information can be adaptively adjusted according to the position information of the target area. For example, the light intensity and light color tone are adjusted, etc., to obtain the second ambient light source information.

[0091] Step 240, perform simulated lighting processing on the second stereoscopic scene map according to the light source parameter information to obtain the third stereoscopic scene map.

[0092] Re-light the second stereoscopic scene information according to the light source parameter information to obtain the third stereoscopic scene information, which can remedy the images with poor effects due to poor light and shadow effects in the user's album. After obtaining the third stereoscopic scene information, the user can be guided to perform virtual shooting on the third stereoscopic scene information to obtain the target image or target video.

[0093] Based on the step of determining the RGB channel information according to the first image involved in step 210, specifically, the first image can be converted into a grayscale image to obtain the grayscale value of its grayscale image and the RGB values of the first image. The ambient light source intensity information is fitted through the RGB brightness values and grayscale values of the whole image. Among them, the ambient light source intensity information may specifically be the RGB channel information.

[0094] The light source parameter information may also include: the RGB channel information. Therefore, in step 240 here, it may specifically include: generating the reconstructed lighting information according to the second ambient light source information and the RGB channel information;

[0095] Perform lighting processing on the second stereoscopic scene information according to the light source position and the reconstructed lighting information to obtain the third stereoscopic scene information.

[0096] Here, considering that the reflection effects of objects made of different materials on light are different, the proportions of different channels in each pixel can be allocated through the channel information of RGB. By generating reconstructed lighting information based on the second ambient light source information and the channel information of RGB, the reconstructed lighting information can express the effects of the light source hitting objects made of different materials. Lighting the second three-dimensional scene information according to the light source position and the reconstructed lighting information can obtain real and natural third three-dimensional scene information.

[0097] In a possible embodiment, step 240 includes:

[0098] Performing simulated lighting processing on the second three-dimensional scene map according to the light source position and the second ambient light source information to obtain a third three-dimensional scene map.

[0099] Combining the position information of the object, referring to the second ambient light source information at the same time, centering on the target area of the object, and controlling the second three-dimensional scene information to perform lighting transformation according to the light source position of the second ambient light source information.

[0100] Step 250, generating a target image or a target video based on the third three-dimensional scene map.

[0101] In a possible embodiment, step 250 may specifically include:

[0102] Receiving the input of the user, where the input is used to control the shooting direction of the virtual camera;

[0103] In response to the input, performing simulated shooting on the third three-dimensional scene map based on the shooting direction to obtain a target image; or,

[0104] Determining multiple shooting directions according to the position information of the object;

[0105] Performing surround shooting on the object in the third three-dimensional scene map based on the multiple shooting directions to obtain multiple video frames;

[0106] Generating a target video based on the multiple video frames.

[0107] On the one hand, the input of the user can be received. The input of the user can be the direction of the user's swipe on the screen of the electronic device, or the movement operation of the user on the electronic device.

[0108] In the case of the movement operation of the user on the electronic device, the gyroscope parameters of the electronic device can be determined according to the input of the user, and the shooting direction of the virtual camera can be controlled according to the gyroscope parameters, so that the reconstructed light source moves towards the target area of the object.

[0109] Among them, a gyroscope is a device that uses the angular momentum of a high-speed rotating body to detect the angular motion of a housing relative to inertial space about one or two axes orthogonal to the axis of rotation. An angular motion detection device made using other principles and having the same function is also called a gyroscope.

[0110] Here, the shooting direction of the virtual camera can be controlled by user input. Since the second ambient light source information in the third three-dimensional scene information is determined, when the shooting direction of the camera changes, different directions of the object will be illuminated by the fixed second ambient light source information, making the whole more three-dimensional. At the same time, the interaction between the user and the electronic device can be combined to create a more real and immersive three-dimensional experience. By simulating the shooting of the third three-dimensional scene map based on the shooting direction, a real and natural target image can be obtained.

[0111] On the other hand, multiple shooting directions can be determined according to the position information of the object; based on the multiple shooting directions, the object in the third three-dimensional scene map is shot around to obtain multiple video frames; based on the multiple video frames, a target video is generated.

[0112] When the user does not interact, the object can be rendered 360 degrees around in the virtual third three-dimensional scene information, and finally a 360-degree surround lighting video effect is obtained.

[0113] In the step of generating the target video, specifically, multiple shooting directions are determined according to the position information of the object, and based on the multiple shooting directions, the object is shot around to obtain multiple video frames, and the target video is generated according to the multiple video frames. By shooting around the object in the third three-dimensional scene information based on the multiple shooting directions, the authenticity of the three-dimensional dynamic effect can be improved, bringing a more interesting creation experience to the user.

[0114] In the embodiment of the present application, by analyzing the acquired first image, the depth information, the first ambient light source information, and the position information of the object in the first image are determined; according to the first ambient light source information, the first three-dimensional scene map constructed based on the depth information is subjected to light elimination processing to obtain a second three-dimensional scene map, which can automatically eliminate the light information carried in the first image, and adaptively reconstruct the light source parameter information according to the depth information, the first ambient light source information, and the position information. Here, since the light source parameter information is determined according to the position information of the object and the first ambient light source information, the light information carried in the first image can be considered, and the object in the first image can be better illuminated, ensuring the authenticity of the reconstructed light source parameter information. By performing simulated lighting processing on the second three-dimensional scene map according to the light source parameter information, a third three-dimensional scene map that is real, natural, and has a high degree of adaptation to the object can be obtained. Finally, based on the third three-dimensional scene map, a real, natural, and highly adaptable target image or target video for the object can be generated.

[0115] In the image processing method provided by the embodiments of the present application, the execution subject may be an image processing device. In the embodiments of the present application, taking the image processing device executing the image processing method as an example, the image processing device provided by the embodiments of the present application is described.

[0116] Figure 4 It is a block diagram of an image processing device provided by the embodiments of the present application. The device 400 includes:

[0117] An analysis module 410, configured to analyze the acquired first image to determine the depth information of the first image, the first ambient light source information, and the position information of the object in the first image.

[0118] An elimination module 420, configured to perform light elimination processing on the first stereoscopic scene map according to the first ambient light source information to obtain a second stereoscopic scene map, where the first stereoscopic scene map is constructed based on the depth information.

[0119] A determination module 430, configured to determine the light source parameter information according to the depth information, the first ambient light source information, and the position information.

[0120] A simulated illumination module 440, configured to perform simulated illumination processing on the second stereoscopic scene map according to the light source parameter information to obtain a third stereoscopic scene map.

[0121] A generation module 450, configured to generate a target image or a target video based on the third stereoscopic scene map.

[0122] In a possible embodiment, the analysis module 410 is specifically configured to:

[0123] Analyze the first image to determine the depth information.

[0124] The analysis module 410 includes:

[0125] An extraction module, configured to extract the position information from the first image according to the depth information.

[0126] A first determination module, configured to determine the first ambient light source information according to the position information.

[0127] In a possible embodiment, the extraction module is specifically configured to:

[0128] Extract a first object image from the first image according to the depth information;

[0129] Perform binarization processing on the first object image to obtain a second object image;

[0130] Perform filtering processing on the first object image to obtain a third object image;

[0131] Merge the second object image and the third object image to obtain a fourth object image;

[0132] Extract position information from the fourth object image.

[0133] In a possible embodiment, the first determination module is specifically configured to:

[0134] Determine the brightness information of each surface of the object from the first image according to the position information;

[0135] Calculate the brightness information based on the law of reflection of light to obtain the first ambient light source information.

[0136] In a possible embodiment, the elimination module 420 is specifically configured to:

[0137] Generate illumination information according to the first ambient light source information;

[0138] Eliminate the illumination information from the first stereoscopic scene map to obtain a second stereoscopic scene map.

[0139] In a possible embodiment, the light source parameter information includes the light source position and the second ambient light source information. The determination module 430 is specifically configured to:

[0140] Determine the light source position according to the position information;

[0141] According to the depth information, determine the position information of the target area closest to the original camera from the position information, where the original camera is the camera that captured the first image;

[0142] Determine the second ambient light source information according to the position information of the target area and the first ambient light source information.

[0143] In a possible embodiment, the simulated illumination module 440 is specifically configured to:

[0144] Perform simulated illumination processing on the second stereoscopic scene map according to the light source position and the second ambient light source information to obtain a third stereoscopic scene map.

[0145] In a possible embodiment, the generation module 450 is specifically configured to:

[0146] Receive the input of the user, where the input is used to control the shooting direction of the virtual camera;

[0147] In response to the input, perform simulated shooting on the third stereoscopic scene map based on the shooting direction to obtain a target image; or,

[0148] Determine multiple shooting directions according to the position information of the object;

[0149] Performing circumferential shooting on the objects in the third three-dimensional scene map based on multiple shooting directions to obtain multiple video frames;

[0150] Generating a target video based on the multiple video frames.

[0151] In the embodiments of the present application, by analyzing the acquired first image, the depth information, the first ambient light source information, and the position information of the object in the first image are determined; according to the first ambient light source information, the first three-dimensional scene map constructed based on the depth information is subjected to light elimination processing to obtain a second three-dimensional scene map, which can automatically eliminate the light information inherent in the first image, and adaptively reconstruct the light source parameter information according to the depth information, the first ambient light source information, and the position information. Here, since the light source parameter information is determined according to the position information of the object and the first ambient light source information, the light information inherent in the first image can be considered, and the object in the first image can be better illuminated, ensuring the authenticity of the reconstructed light source parameter information. Performing simulated light processing on the second three-dimensional scene map according to the light source parameter information can obtain a third three-dimensional scene map that is real, natural, and has a high degree of adaptation to the object in terms of lighting. Finally, based on the third three-dimensional scene map, a target image or a target video that is real, natural, and has a high degree of adaptation to the object in terms of lighting can be generated.

[0152] The image processing device in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0153] The image processing device according to an embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0154] The image processing device provided by the embodiments of the present application can implement each process implemented by the above method embodiments. To avoid repetition, it will not be elaborated here.

[0155] Optionally, as Figure 5 shown, an embodiment of the present application further provides an electronic device 510, including a processor 511, a memory 512, a program or instruction stored on the memory 512 and executable on the processor 511. When the program or instruction is executed by the processor 511, it implements each step of any of the above image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0156] It should be noted that the electronic device in the embodiments of the present application includes the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0157] Figure 6 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0158] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610, etc.

[0159] Those skilled in the art can understand that the electronic device 600 may further include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 610 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 6 The structure of the electronic device shown in does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0160] Among them, the processor 610 is used to analyze the acquired first image to determine the depth information of the first image, the first ambient light source information, and the position information of the object in the first image.

[0161] The processor 610 is further used to perform light elimination processing on the first stereoscopic scene map according to the first ambient light source information to obtain a second stereoscopic scene map, and the first stereoscopic scene map is constructed based on the depth information.

[0162] The processor 610 is further configured to determine light source parameter information according to the depth information, the first environmental light source information, and the position information.

[0163] The processor 610 is further configured to perform simulated lighting processing on the second stereoscopic scene map according to the light source parameter information to obtain a third stereoscopic scene map.

[0164] The processor 610 is further configured to generate a target image or a target video based on the third stereoscopic scene map.

[0165] Optionally, the processor 610 is further configured to analyze the first image to determine the depth information;

[0166] Extract the position information from the first image according to the depth information.

[0167] Determine the first environmental light source information according to the position information.

[0168] Optionally, the processor 610 is further configured to extract a first object image from the first image according to the depth information;

[0169] Perform binarization processing on the first object image to obtain a second object image;

[0170] Perform filtering processing on the first object image to obtain a third object image;

[0171] Merge the second object image and the third object image to obtain a fourth object image;

[0172] Extract the position information from the fourth object image.

[0173] Optionally, the processor 610 is further configured to determine the brightness information of each surface of the object from the first image according to the position information;

[0174] Calculate the first environmental light source information based on the law of reflection of light for the brightness information.

[0175] Optionally, the processor 610 is further configured to generate lighting information according to the first environmental light source information;

[0176] Eliminate the lighting information from the first stereoscopic scene map to obtain a second stereoscopic scene map.

[0177] Optionally, the processor 610 is further configured to determine the light source position according to the position information;

[0178] Determine the position information of the target area closest to the original camera from the position information according to the depth information, where the original camera is the camera that captured the first image;

[0179] Determine the second environmental light source information according to the position information of the target area and the first environmental light source information.

[0180] Optionally, the processor 610 is further configured to perform simulated illumination processing on the second three-dimensional scene graph according to the light source position and the second ambient light source information to obtain a third three-dimensional scene graph.

[0181] Optionally, the processor 610 is further configured to receive an input from a user, where the input is used to control the shooting direction of the virtual camera;

[0182] In response to the input, perform simulated shooting on the third three-dimensional scene graph based on the shooting direction to obtain a target image; or,

[0183] Determine a plurality of shooting directions according to the position information of the object;

[0184] Perform surround shooting on the object in the third three-dimensional scene graph based on the plurality of shooting directions to obtain a plurality of video frames;

[0185] Generate a target video based on the plurality of video frames.

[0186] In the embodiments of the present application, by analyzing the acquired first image, the depth information of the first image, the first ambient light source information, and the position information of the object in the first image are determined; according to the first ambient light source information, illumination elimination processing is performed on the first three-dimensional scene graph constructed based on the depth information to obtain a second three-dimensional scene graph, which can automatically eliminate the illumination information inherent in the first image, and adaptively reconstruct the light source parameter information according to the depth information, the first ambient light source information, and the position information. Here, since the light source parameter information is determined according to the position information of the object and the first ambient light source information, the illumination information inherent in the first image can be considered, and the object in the first image can be better illuminated, ensuring the authenticity of the reconstructed light source parameter information. Performing simulated illumination processing on the second three-dimensional scene graph according to the light source parameter information can obtain a third three-dimensional scene graph that is real and natural and has a high degree of adaptation to the object. Finally, based on the third three-dimensional scene graph, a target image or a target video that is real, natural, and has a high degree of adaptation to the object can be generated.

[0187] It should be understood that in the embodiments of the present application, the input unit 604 may include a Graphics Processing Unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes the image data of static pictures or video images obtained by an image capture device (such as a camera) in a video image capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of, for example, a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also referred to as a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. The other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here. The memory 609 may be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 610 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interfaces, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 610.

[0188] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both a volatile and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 609 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0189] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 610 either.

[0190] The embodiments of the present application also provide a readable storage medium, on which a program or instructions are stored. When the program or instructions are executed by a processor, each process of the above embodiment of the image processing method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0191] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0192] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above embodiment of the image processing method and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0193] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0194] The embodiments of the present application provide a computer program product. The program product is stored in a storage medium and is executed by at least one processor to implement each process of the above embodiment of the image processing method and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0195] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed. It may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described method may be executed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0196] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0197] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Analyze the acquired first image to determine the depth information of the first image, the first environmental light source information, and the position information of the object in the first image; According to the first environmental light source information, perform light elimination processing on the first three-dimensional scene graph to obtain a second three-dimensional scene graph, where the first three-dimensional scene graph is constructed based on the depth information; Determine the light source parameter information according to the depth information, the first environmental light source information, and the position information; Perform simulated light processing on the second three-dimensional scene graph according to the light source parameter information to obtain a third three-dimensional scene graph; Generate a target image or a target video based on the third three-dimensional scene graph; Among them, eliminating the light information in the first three-dimensional scene information to obtain the second three-dimensional scene graph includes: placing the entire three-dimensional scene in a situation without lighting to obtain the second three-dimensional scene graph; Among them, the light source parameter information includes the light source position and the second environmental light source information. Determining the light source parameter information according to the depth information, the first environmental light source information, and the position information includes: Determine the light source position according to the position information; according to the depth information, determine the position information of the target area closest to the original camera in the position information, where the original camera is the camera that captured the first image; determine the second environmental light source information according to the position information of the target area and the first environmental light source information.

2. The method according to claim 1, wherein Analyzing the acquired first image to determine the depth information of the first image, the first environmental light source information, and the position information of the object in the first image includes: Analyze the first image to determine the depth information; Extract the position information from the first image according to the depth information; Determine the first environmental light source information according to the position information.

3. The method according to claim 2, wherein Extracting the position information from the first image according to the depth information includes: Extract the first object image from the first image according to the depth information; Perform binarization processing on the first object image to obtain a second object image; Perform filtering processing on the first object image to obtain a third object image; Merge the second object image and the third object image to obtain a fourth object image; Extract the position information from the fourth object image.

4. The method according to claim 2, wherein Determining the first environmental light source information according to the position information includes: Determine the brightness information of each surface of the object from the first image according to the position information; Calculate the first environmental light source information based on the law of reflection of light for the brightness information.

5. The method according to claim 1, wherein Performing light elimination processing on the first three-dimensional scene graph according to the first environmental light source information to obtain a second three-dimensional scene graph includes: Generate light information according to the first environmental light source information; Eliminate the light information from the first three-dimensional scene graph to obtain the second three-dimensional scene graph.

6. The method according to claim 1, wherein Performing simulated light processing on the second three-dimensional scene graph according to the light source parameters to obtain a third three-dimensional scene graph includes: Performing simulated lighting processing on the second three-dimensional scene graph according to the light source position and the second ambient light source information to obtain the third three-dimensional scene graph.

7. The method according to claim 1, characterized in that Generating a target image or a target video based on the third three-dimensional scene graph includes: Receiving an input from a user, where the input is used to control the shooting direction of a virtual camera; In response to the input, performing simulated shooting on the third three-dimensional scene graph based on the shooting direction to obtain the target image; or, Determining a plurality of shooting directions according to the position information of the object; Performing surround shooting on the object in the third three-dimensional scene graph based on the plurality of shooting directions to obtain a plurality of video frames; Generating the target video based on the plurality of video frames.

8. An image processing apparatus, characterized in that, The apparatus includes: An analysis module for analyzing the acquired first image to determine the depth information of the first image, the first ambient light source information, and the position information of the object in the first image; An elimination module for performing light elimination processing on the first three-dimensional scene graph according to the first ambient light source information to obtain a second three-dimensional scene graph, where the first three-dimensional scene graph is constructed based on the depth information; eliminating the light information in the first three-dimensional scene information to obtain a second three-dimensional scene graph includes: placing the entire three-dimensional scene in a situation without lighting to obtain the second three-dimensional scene graph; A determination module for determining light source parameter information according to the depth information, the first ambient light source information, and the position information; The light source parameter information includes a light source position and second ambient light source information. Determining the light source parameter information according to the depth information, the first ambient light source information, and the position information includes: Determining the light source position according to the position information; according to the depth information, determining the position information of the target area closest to the original camera from the position information, where the original camera is the camera that captured the first image; determining the second ambient light source information according to the position information of the target area and the first ambient light source information; A simulated lighting module for performing simulated lighting processing on the second three-dimensional scene graph according to the light source parameter information to obtain a third three-dimensional scene graph; A generation module for generating a target image or a target video based on the third three-dimensional scene graph.

9. An electronic device, characterized in that, Including a processor and a memory, where the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images