Related methods and related products for rendering virtual objects based on brightness estimation

Abstract

A method for rendering a virtual object based on brightness estimation and a related product are provided. The method for rendering a virtual object based on brightness estimation comprises: capturing an image, wherein at least one object in the image is located on at least one plane; extracting a foreground of the image; estimating the brightness corresponding to the extracted foreground; and rendering the virtual object based on the estimated brightness. Through the embodiments of the present application, the brightness conditions of the real world can be estimated in real time, and the virtual object can be rendered based on the estimated brightness.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 967,739, filed on January 30, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of augmented reality technology, and in particular to a method for rendering a virtual object based on brightness estimation, a method for training a neural network, and related products.

[0004] Prospect Technology

[0005] Augmented reality (AR) applications aim to provide a realistic fusion between the real world and virtual objects. Augmented reality devices can be configured to display augmented reality images to present the illusion that virtual objects exist in real-world physical space. An important factor for realistic AR is to correctly estimate the brightness of the real world.

[0006] Real-world brightness estimation is a challenging problem. Currently, typical solutions to this problem rely on inserting objects with known geometry and / or reflective properties in the real world (e.g., light probes). However, having to insert a known object in the real world is restrictive and therefore not easily adaptable to practical applications. Summary of the invention

[0007] Embodiments of the present application provide a method for rendering a virtual object based on brightness estimation, a method for training a neural network, and related products, which can estimate the brightness conditions of the real world in real time and be used to render virtual objects based on the estimated brightness.

[0008] In a first aspect, a method for rendering a virtual object based on brightness estimation is provided, the method for rendering a virtual object based on brightness estimation comprising the following steps: capturing an image, wherein at least one object in the image is located on at least one plane; extracting a foreground of the image; estimating the brightness corresponding to the extracted foreground; and rendering the virtual object based on the estimated brightness.

[0009] In a second aspect, a method for training a neural network is provided, the method for training a neural network comprising: obtaining a training image from a first data set, the first data set comprising a plurality of images and ground truth parameters corresponding to each image, the training image being any image included in the first data set; extracting a foreground of the training image; inputting the extracted foreground of the training image into a neural network to obtain prediction parameters; rendering an image with the prediction parameters; calculating a loss of the neural network based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image; and training the neural network based on the loss.

[0010] In a third aspect, a device for a method of rendering a virtual object based on brightness estimation is provided, and the device for the method of rendering a virtual object based on brightness estimation includes a shooting unit, an extraction unit, an estimation unit, and a rendering unit. The shooting unit is used to shoot an image, and at least one object in the image is located on at least one plane. The extraction unit is used to extract a foreground of the image. The estimation unit is used to estimate the brightness corresponding to the extracted foreground. The rendering unit is used to render the virtual object based on the estimated brightness.

[0011] In a fourth aspect, a device for obtaining a training neural network is provided, comprising: an acquisition unit, an extraction unit, an input unit, a rendering unit, a calculation unit and a training unit. The acquisition unit is used to obtain a training image from a first data set, the first data set comprising a plurality of images and ground truth parameters corresponding to each image, and the training image is any image included in the first data set. The extraction unit is used to extract the foreground of the training image. The input unit is used to input the extracted foreground of the training image into the neural network to obtain prediction parameters. The rendering unit is used to render an image using the prediction parameters. The calculation unit is used to calculate the loss of the neural network based on the prediction parameters, the rendered image, the training image and the ground truth parameters corresponding to the training image. The training unit is used to train the neural network based on the loss.

[0012] In a fifth aspect, a terminal device is provided, comprising a processor and a memory for storing one or more programs, wherein the one or more programs are used to be executed by the processor, and the one or more programs include instructions for executing part or all of the operations in the method described in the first aspect or the second aspect.

[0013] In a sixth aspect, a non-transitory computer-readable storage medium is provided for storing computer programs for electronic data exchange, wherein one or more programs include instructions for executing part or all of the operations in the method described in the first aspect or the second aspect.

[0014] In a seventh aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program enabling a computer to execute part or all of the operations in the method described in the first aspect or the second aspect.

[0015] In an embodiment of the present application, an image in a real scene is captured, at least one object in the image is located on at least one plane, the foreground of the image is extracted, the brightness corresponding to the extracted foreground is estimated, and finally the virtual object is rendered based on the estimated brightness. Therefore, through the input of the trained neural network and the image containing one or more objects placed on the plane area in the captured real scene, the brightness conditions of the real world can be estimated in real time, and the virtual object can be rendered in the real scene with the estimated brightness, thereby improving the rendering quality and realism of the virtual object rendered in the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the application, the drawings required for use in the implementation will be briefly introduced below. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a schematic diagram of an exemplary operating environment according to an embodiment.

[0018] Figure 2 is a schematic diagram of the structure of an exemplary neural network architecture according to an embodiment.

[0019] Figure 3 is a schematic diagram of rendering a virtual object in a real scene using estimated brightness according to an embodiment.

[0020] Figure 4 is a flowchart of a method for rendering a virtual object based on brightness estimation according to an embodiment.

[0021] Figure 5(a)-5(d) is a schematic diagram of an exemplary foreground extraction process according to an embodiment.

[0022] Figure 6(a)-6(b) is a schematic diagram of an exemplary neural network design according to an embodiment.

[0023] Figure 7 is a schematic diagram of an exemplary ground truth data generation process.

[0024] Figure 8 It is a schematic diagram comparing brightness prediction and normal map prediction of a neural network trained based on a plane normal map and a neural network not trained based on a plane normal map according to an embodiment.

[0025] Fig. 9 It is a schematic diagram comparing brightness prediction and normal map prediction of a neural network trained based on a plane normal map (middle) and a neural network not trained based on a plane normal map (upper) according to other embodiments.

[0026] Fig.10 It is a comparison chart of brightness estimation of the prior art and the present application.

[0027] Fig.11 is a schematic diagram of an exemplary rendering of a virtual object in a real scene.

[0028] Fig.12 is a flowchart of a method for training a neural network according to an embodiment.

[0029] Fig.13 is a structural diagram of an apparatus for rendering a virtual object based on brightness estimation according to an embodiment.

[0030] Fig.14 is a schematic structural diagram of an apparatus for training a neural network according to an embodiment.

[0031] Fig.15 is a schematic structural diagram of a terminal device according to an embodiment.

[0032] Fig.16 is a schematic diagram of the structure of a terminal device according to other embodiments. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the present invention rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0034] The terms "first" and "second" and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] The terminal devices involved in the embodiments of the present invention may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem, as well as various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. For the convenience of description, the above-mentioned devices are collectively referred to as terminal devices.

[0037] In order to facilitate a better understanding of the embodiments of the present application, the relevant technologies involved in the present application are briefly introduced below.

[0038] Brightness estimation has been a long-standing problem in computer vision and graphics. A straightforward way to estimate the brightness of an environment is to use physical light probes to capture the radiance of light at the target location. Photos of a specular sphere at different exposures can be used to compute the brightness at the location of the sphere. In addition to specular spheres, brightness can also be estimated using hybrid spheres, known 3D objects, objects of known surface material, and even human faces as representatives of light probes. However, the process of manually capturing high-quality brightness maps is expensive and difficult to generalize, especially when the goal is training data on a dense set of visible locations obtained in a variety of environments.

[0039] Another approach to estimating brightness is to comprehensively optimize the scene's geometry, reflectance properties, and brightness model in order to find a set of values ​​that better explain the input image. However, directly optimizing all scene parameters is often a highly under-constrained problem, i.e., an error in one parameter estimate can easily propagate into another parameter estimate. Therefore, to simplify the optimization process, many existing methods either assume additional user-provided ground truth information as input or make strong assumptions about the brightness model.

[0040] Deep learning has recently shown good results on many computer vision tasks, including depth estimation and intrinsic image decomposition. Recently, LeGendre et al. proposed to describe the brightness estimation function as an end-to-end neural network. LeGendre et al. used balls with different bidirectional reflectance distribution functions (BRDFs) as brightness probes and used Google Pixel phones to capture images. In order to achieve real-time performance, LeGendre et al. compressed the environment to a very small size (32x32). This makes the prediction map unstable and overly sensitive to lighting intensity, which will destroy temporal consistency. In terms of high precision, Shuran et al. proposed an end-to-end network that directly maps LDR images to HDR environment maps using geometric winding and adversarial networks. However, the amount of calculation required to obtain HDR environment maps is large and difficult to perform in real time. Moreover, using HDR maps as brightness will also increase rendering costs. Augmented reality (AR) applications require real-time or near real-time. In an embodiment of the present application, the fifth-order spherical harmonic brightness is used to approximate the environment map. Without losing too much accuracy, some parameters can be used instead of HDR maps to reduce prediction and rendering costs.

[0041] With the development of the deep learning community, many tasks are becoming bridges between computer vision (2D) and computer graphics (3D), and brightness estimation is one of them. Differentiable renderers have been widely used in tasks such as joint reconstruction of shape and BRDF, joint estimation of brightness and BRDF, etc. In order to link changes in captured images with changes in 3D shape control, many existing technologies have utilized derivatives of rendering. However, these derivatives of rendering are designed based on modern complex rendering systems (rasterization or ray tracing) and are too heavy in deep learning systems. Another problem is that these derivatives of rendering require all three-dimensional geometric attributes (coordinates, normals, BRDFs) to perform accurate rendering. In such a system, reconstruction tasks are always intertwined, and to reconstruct one attribute, all other attributes need to be reconstructed as a "side effect". However, it is not a good idea to reconstruct all attributes at the same time. In terms of supervised learning, 3D annotation is too expensive and often inaccurate. In terms of unsupervised learning, the current amount of data is not enough to learn the intrinsic manifold between attributes. Therefore, in an embodiment of the present application, a lightweight differentiable screen space renderer is developed. While it still requires several attributes (brightness (e.g. SH coefficient), normal, reflectivity, and roughness) to render, there are far fewer. All of these attributes are defined per pixel and will be cheaper to render, thus providing a more computationally efficient solution.

[0042] A detailed description of the embodiments of the present application will be given below.

[0043] Figure 1 FIG. 1 is a schematic diagram of an exemplary operating environment according to an embodiment. Figure 1 As shown, the exemplary operating environment 100 includes at least two computing devices interconnected by one or more networks 11. One or more networks 11 allow a computing device to connect to another computing device and communicate with it. In some embodiments, the at least two computing devices include a terminal device 13 and a server 12. The at least two computing devices may include other computing devices not shown, which are not limited here. One or more networks 11 may include a secure network such as an enterprise private network, an unsecure network such as a wireless open network, a local area network (LAN), a wide area network (WAN) and the Internet. For ease of explanation below, one network 11 is used.

[0044] In some embodiments, the terminal device 13 includes a network interface 121, a processor 122, a memory 123, a camera 124, a sensor 125, and a display 126 that communicate with each other. The network interface 121 allows the terminal device 13 to connect to the network 11. The network interface 121 may include at least one of a wireless network interface, a modem, or a wired network interface. The processor 122 allows the terminal device 13 to execute computer-readable instructions stored in the memory 123 to perform the processes discussed herein. The camera 124 can capture color images and / or depth images of the environment. The terminal device 13 may include a rear camera that captures images of the environment and a front camera that captures images of the terminal user of the terminal device. The sensor 125 may generate motion and / or orientation information associated with the terminal device 12. In some cases, the sensor 125 may include an inertial measurement unit (IMU). The display 126 may display digital images and / or videos. The display 126 may include a perspective display. The display 126 may include a light emitting diode (LED) or an organic LED (OLED) display.

[0045] In some embodiments, various components of the terminal device 13, such as the network interface 121, the processor 122, the memory 123, the camera 124, and the sensor 125, can be integrated on a single chip substrate. In one example, the network interface 121, the processor 122, the memory 123, the camera 124, and the sensor 125 can be integrated into a system on a chip (SOC). In other embodiments, the network interface 121, the processor 122, the memory 123, the camera 124, and the sensor 125 can be integrated in a single package.

[0046] In some embodiments, the terminal device 13 can provide a natural user interface (NUI) by using the camera 124, the sensor 125, and the gesture recognition software running on the processor 122. Using the natural user interface, the body parts and movements of a person can be detected, interpreted, and used to control various aspects of the computing application in the terminal. In one example, a computing device using a natural user interface can infer the intention of a person interacting with the computing device (for example, the terminal user performs a specific gesture to control the computing device).

[0047] In one example, the terminal device 13 includes a head mounted display (HMD) that provides an augmented, mixed, virtual reality environment or a mixed reality environment to the terminal user of the HMD. In the case of an augmented or mixed reality environment, the HMD may include a video see-through and / or optical see-through system. The optical see-through HMD worn by the terminal user may allow actual direct viewing of the real world environment (e.g., through a transparent lens), and at the same time may project an image of a virtual object into the terminal user's field of view, thereby enhancing the real world environment perceived by the terminal user with the virtual object.

[0048] Using the HMD, the terminal user can wear the HMD to move in a real-world environment (e.g., a living room) and perceive a view of the real world superimposed by the image of the virtual object. The virtual object can appear to have a consistent spatial relationship with the real-world environment, that is, when the terminal user turns his head or moves in the real-world environment, the image displayed to the terminal user will change, so that the virtual object appears to exist in the real-world environment perceived by the terminal user. The virtual object can also appear to be fixed relative to the terminal user's perspective, for example, no matter how the terminal user turns his head or moves in the real-world environment, the virtual menu always appears in the upper right corner of the terminal user's perspective. In one embodiment, the environmental mapping of the real-world environment can be performed by the server 12 (i.e., on the server side), and the camera positioning can be performed on the terminal device 13 (i.e., on the client side). The virtual object can include a text description associated with the real-world object.

[0049] In some embodiments, a terminal device such as terminal device 13 can communicate with a cloud server such as server 12, and can provide the server with location information associated with the terminal device (e.g., the location of the terminal device by means of GPS coordinates) and / or image information (e.g., information about objects detected in the field of view of the terminal device). In response, the server can send one or more virtual objects to the terminal device based on the location information and / or image information provided to the server. In one embodiment, the terminal device 13 can specify a specific file format for receiving one or more virtual objects, and the server 12 can send one or more virtual objects to the terminal device 13, the one or more virtual objects being contained in a file of the specific file format.

[0050] Figure 2 is a schematic diagram of an exemplary neural network architecture according to an embodiment. Figure 2 As shown, for a single input image, the initial foreground and enhanced foreground are extracted. The enhanced foreground is fed into the initial encoder and then decoded into normal, reflectivity, and roughness respectively, and then the fifth-order spherical harmonic brightness is regressed. The mask of the target object is then multiplied by the predicted normal, reflectivity, and roughness to remove the planar area. The screen space renderer will take the normal, reflectivity, roughness, and SH brightness to generate a re-rendered scene image.

[0051] Figure 3 This is a diagram of rendering virtual objects in real scenes using estimated brightness. On the left is an indoor image captured by a mobile device such as a mobile phone, and then the brightness conditions are estimated based on the captured image, and the virtual objects are rendered into the real scene. Some detailed rendering effects such as soft shadows and smooth surfaces are shown in the enlarged image.

[0052] Figure 4 is a flow chart of a method for rendering a virtual object based on brightness estimation according to an embodiment. The method for rendering a virtual object based on brightness estimation may be applicable to Figure 1 The terminal device 13 shown in FIG. In this embodiment, an image in a real scene is captured in real time, at least one object in the image is located on at least one plane, and then the foreground of the image is extracted, and then the brightness corresponding to the extracted foreground is estimated, and finally the virtual object is rendered in the real scene with the estimated brightness. Therefore, through the input of the trained neural network and the image captured in the real scene containing one or more objects placed on a plane area, the brightness conditions of the real world can be estimated in real time, and the virtual object can be rendered in the real scene with the estimated brightness, thereby improving the rendering quality and realism of the virtual object rendered in the real scene.

[0053] like Figure 4 As shown, the method for rendering a virtual object based on brightness estimation includes the following contents.

[0054] At block 402, an image is captured, wherein at least one object in the image is located on at least one plane.

[0055] Specifically, the brightness estimation is based on the visual appearance of at least one object in a real scene, and the image can be captured by a monocular RGB camera of the terminal device.

[0056] At block 404, the foreground of the image is extracted.

[0057] The background content of the image is usually too small, and the area near the edge of the image may even be distorted due to camera projection. Therefore, the foreground of the image is extracted to reduce the computational complexity.

[0058] In one embodiment, the foreground of an image can be extracted as follows. At least one preset object is detected in the image. An object located at the center of the image is selected from the at least one preset object as a target object. The foreground of the image is determined and extracted based on the target object and the plane where the target object is located. As shown in Figures 5(a)-(d), Figures 5(a)-(d) show an exemplary foreground extraction process according to an embodiment, wherein Figure 5(a) is a schematic diagram of an exemplary captured image, Figure 5(b) is a schematic diagram of a target object in an image, Figure 5(c) is a schematic diagram of an initial extraction (e.g., segmentation), and Figure 5(d) is a schematic diagram of a final extraction.

[0059] Specifically, preset objects (e.g., bananas, vases, etc.) that are most likely to be found on indoor tables or the ground are collected from the common objects in context (COCO) dataset. Figure 5(a) shows such an example. In order to perform foreground extraction, Detectron 2 is used to detect at least one preset object and a target object. As shown in Figure 5(a), at least one preset object detected in the image may include a sofa, a table, and a flower pot. After detecting the at least one preset object, only the object located in the center of the image is selected from the at least one preset object as the target object.

[0060] In one embodiment, an object located at the center of the image as the target object can be selected from at least one preset object as follows. An object located at least partially approximately at the center of the image is selected from at least one preset object as the target object. In other words, an object in which a first percentage of pixels are located at a second percentage of the center of the image is selected from at least one preset object as the target object. For example, an object in which 95% of the segmented pixels are located at 70% of the center of the image is selected as the target object. FIG5(b) shows such an example. As shown in FIG5(b), only the sofa is selected as the target object.

[0061] In one embodiment, the foreground of the image is determined based on the target object and the plane in which the target object is located, as described below.

[0062] A bounding box is determined based on the target object. The bounding box is divided into an upper bounding box and a lower bounding box. The lower bounding box is expanded by a magnification factor to include a portion of the plane where the target object is located. The portion of the image surrounded by the upper bounding box and the expanded lower bounding box is determined to be the foreground of the image. In other words, the target object and the portion of the plane located in the expanded second bounding box are determined to be the foreground of the image.

[0063] Specifically, a bounding box is determined based on the mask of the target object. As shown in Figure 5(c), the lower half of the bounding box is located and expanded in the x and y directions according to a preset magnification (e.g., 1.3). Therefore, a portion of the plane where the target object is located is almost entirely included in the expanded area. The final enhanced segmentation, i.e., the foreground of the image, is shown in Figure 5(d).

[0064] At block 406, the brightness corresponding to the extracted foreground is estimated.

[0065] Specifically, one input of the neural network is the extracted foreground IA. Figure 2 As shown, the mask of the target object (as shown in Figure 5(c)) is denoted as M, and the enhanced mask (as shown in Figure 5(d)) is denoted as M A The mask of the plane can be expressed as M P , and M P =M A -M. SceneParser(·) represents a neural network consisting of an encoder block and a decoder block, and then gives the predicted normal map by the following method Reflectivity Roughness and spherical harmonics (SH) coefficients (Use ~ to distinguish from ground truth parameters):

[0066]

[0067] The output of an existing AR framework (e.g., ARCore) for the input image is taken as N P All N discussed here P All are in screen space coordinates.

[0068] Since the decoder directly outputs normal, reflectivity and roughness with a plane attached, the mask M of the target object is applied to get the predicted normal map Predicted reflectivity Predicted roughness exist Figure 2 , predicting the normal map Predicted reflectivity Predicted roughness They are (e), (g) and (i) respectively.

[0069] 6(a) and 6(b) are schematic diagrams of exemplary neural network designs according to an embodiment. As an example, the neural network may include a brightness prediction module named SHE_st(·) and an initial encoder named InitEncoder(·). SHE_st(·) is connected after InitEncoder(·). SHE_st(·) contains two fully connected layers that regress 36 spherical harmonic coefficients for each color channel. The predicted spherical harmonic (SH) coefficients It can be obtained by the following formula:

[0070]

[0071] In addition, the neural network can also include three sub-autoencoders for normal, reflectivity and roughness, and these three sub-autoencoders share the same encoder initecoder(·) and have their own decoders, named NormalDecoder(·), AlbedoDecoder(·), and RoughDecoder(·). The detailed architecture design is shown in Figure 6(A). Predicted normal map Reflectivity Roughness It can be obtained by the following formula:

[0072]

[0073] represents the inner product of two images. By taking the inner product of M and the output of the decoder, the plane can be removed.

[0074] In order to predict the brightness in real scenes, a large receptive field is required. Therefore, for example, InitEncoder(·) can have 6 convolutional layers with a stride of 2, so that each pixel of the output can be affected by the entire image. For NormalDecoder(·), AlbedoDecoder(·), and RoughDecoder(·), except that NormalDecoder(·) has an additional input NP, they use transposed convolution for decoding and add skip links to recover more details.

[0075] FIG6(a) shows the detailed structure of such an encoder-decoder pair, and FIG6(b) shows the detailed structure of SHE_st(·).

[0076] At block 408 , the virtual object is rendered with the estimated brightness.

[0077] In one embodiment, the brightness corresponding to the extracted foreground may be estimated as follows: The extracted foreground is input into a neural network. A predicted SH coefficient output by the neural network is obtained. The brightness corresponding to the image is determined based on the predicted SH coefficient.

[0078] Specifically, high-level AR frameworks such as ARCore or ARKit are often used to provide robust and accurate plane detection. Virtual objects can be rendered on planes in the real scene with estimated brightness.

[0079] It can be seen that in the embodiment of the present application, an image containing at least one object located on at least one plane in a real scene is captured in real time, and then the foreground of the image is extracted, and then the brightness corresponding to the extracted foreground is estimated, and finally the virtual object is rendered in the real scene with the estimated brightness. Therefore, by using a trained neural network and an image containing one or more objects placed on a plane area captured in a real scene as input, the brightness conditions of the real world can be estimated in real time, and the virtual object can be rendered in the real scene with the estimated brightness, thereby improving the rendering quality and realism of the virtual object rendered in the real scene.

[0080] In one embodiment, the method further includes the following: training a neural network with a first data set, wherein the first data set includes a plurality of images and ground truth parameters corresponding to each image, each image includes an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, a ground truth SH coefficient, a ground truth reflectivity, and a ground truth roughness.

[0081] In one embodiment, the method further includes constructing a first data set. Specifically, a plurality of images are selected from a second data set, wherein each of the plurality of images contains an object and a plane on which the object is located, wherein the second data set includes a plurality of panoramic images and a ground truth HDR brightness map corresponding to each of the plurality of panoramic images. For each selected image: a ground truth normal map of the object and a ground truth normal map of the plane are calculated; a ground truth SH coefficient corresponding to the image is extracted by applying spherical convolution to the ground truth HDR brightness map; and a ground truth reflectivity and a ground truth roughness corresponding to the image are calculated. The first data set is constructed based on the selected plurality of images and the ground truth parameters corresponding to the selected plurality of images, wherein the ground truth parameters corresponding to each selected image include the ground truth normal map of the object, the ground truth normal map of the plane, the ground truth SH coefficient, the ground truth reflectivity, and the ground truth roughness.

[0082] To support the training of neural networks, a large database of indoor images and their corresponding normal maps is required. However, obtaining a large database including ground truth normal maps for training is challenging. On the one hand, using physically based rendering engines it is straightforward to synthesize rendered images and normal maps simultaneously, but neural networks trained with synthetic data often perform poorly on real data. On the other hand, existing normal map datasets provide normal maps that are reconstructed in an expensive photometric stereo setting, and thus the amount of data is too small for adequate network training.

[0083] In one embodiment, the first dataset can be constructed based on the existing second dataset (i.e., Matterport3D dataset). Matterport3D contains 194,400 registered HDR RGB-D images, consisting of 10,800 panoramic images located in 90 indoor scenes of different building scales. Specifically, the first dataset is generated using multiple HDR RGB-D images in the Matterport3D dataset. The panoramic image can provide a ground truth HDR brightness map, and the 5th order SH coefficients of these brightness maps are extracted using spherical convolution as the ground truth SH coefficients L gt .

[0084] First, images containing objects and the planes on which the objects lie are selected from the Matterport3D dataset. A plane is defined as: (a) a horizontal plane (nz>(cosπ / 8)); (b) a semantic label, such as floor, furniture, or (c) one or more objects on it. Then, for each image I, the object is transferred to the screen space (e.g., camera coordinates), and the normal maps of the object and the plane are calculated. In addition, spherical convolution is used to extract the 5th-order SH coefficients of the ground truth HDR brightness map as the ground truth SH coefficient L gt In this way, the SH light is rotated to the screen space and the SH coefficients are transformed accordingly. To this end, the necessary labels are constructed for the image I: the ground truth SH coefficients L gt , the ground truth normal map N of the object gt and the ground truth normal map N of the plane p . A total of about 109042 {I,L gt ,N gt ,N p}sample. Figure 7 An exemplary generation process of ground truth data is shown. Figure 7 As shown in Figure 2, the ground truth SH coefficients and normal maps of images containing support planes in the Matterport3D dataset are generated. In the Matterport3D dataset, the ground truth brightness is a panoramic environment map, and the ground truth SH brightness is generated based on the 5th order SH coefficients.

[0085] During the training phase, the normal map N of the plane is extracted directly from the ground truth data in the Matterport3D dataset p In practical applications, the output of an AR framework (such as ARCore) on the input image can be used as the normal map N of the plane. p All N discussed here p Both can be in screen space coordinates.

[0086] In one embodiment, a neural network is trained using a first data set, as described below.

[0087] A training image is obtained from a first data set, wherein the training image is any image in the first data set. A foreground of the training image is extracted. The extracted foreground of the training image is input into a neural network to obtain prediction parameters, wherein the prediction parameters include a predicted normal map of an object in the training image, a predicted normal map of a plane in the training image, a predicted SH coefficient, a predicted reflectivity, and a predicted roughness. An image is rendered using the predicted parameters. A loss of the neural network is calculated based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image. The neural network is trained based on the loss.

[0088] In this embodiment, adding the plane normal map to the brightness estimation and neural network training helps to improve the accuracy of brightness estimation. As shown in Table 1, for the same training image, the neural network is trained with and without the plane normal map. P The recovery error reported by train 1 is significantly larger than that of train S (with N P It can be seen that the normal map of the plane has a significant impact on the brightness. Figure 8 It is a schematic diagram comparing brightness prediction and normal map prediction of a neural network trained based on a plane normal map and a neural network not trained based on a plane normal map according to an embodiment. Figure 8 The quantitative comparison shows that in the absence of N P Under the guidance of , the texture on the object may mislead the normal map estimation and further produce incorrect brightness estimation.

[0089] Table 1: Quantitative comparison of images

[0090] S SP <![CDATA[5th-order SH coefficient (10 -2 )]]> 9.833 3.746 <![CDATA[Normal map (10 -2 )]]> 6.591 4.184

[0091] For multiple images taken by a mobile phone, N P How this affects brightness or brightness estimation has been demonstrated. Fig. 9FIG. 1 is a schematic diagram comparing brightness prediction and normal map prediction of a neural network trained based on a plane normal map (middle) and a neural network not trained based on a plane normal map (upper) according to other embodiments. Fig. 9 As shown, in the top example, the normal map output from SP is incorrect, part of the box means that the normal map on that area is very different, which is incorrect. In the bottom instance, the normal map output from SP is too smooth. In both examples, the lighting is too dark, which is not good for the SP result.

[0092] In one example, an image is rendered with the predicted parameters, as described below.

[0093] Since the embodiment of the present application does not include the transmission effect and self-illumination, the rendering equation can be simplified to the reflection equation so as to render with the output of the neural network:

[0094]

[0095] L 0 Indicates that the object is located at a specific position Outward in the direction of the eyes Total spectral radiation, L i Indicates along Direction The incident radiation at r express The bidirectional reflectance distribution function (BRDF) at . This integral is the sum of the normal-oriented hemisphere Ω towards Normal map at

[0096] brightness

[0097] The brightness is parameterized as a fifth-order spherical harmonic brightness. Since the fifth-order spherical harmonic brightness is a global brightness, the radiance depends only on the direction. More specifically, we can express L as i :

[0098]

[0099] (θ i ,φ i ) represent the altitude and azimuth in the camera coordinate system, respectively. express The predicted SH coefficient in Y lm represents the spherical harmonic basis.

[0100] BRDF

[0101] In the microfacet BRDF model Based on this, the BRDF model is defined as:

[0102]

[0103] D, F, and G represent normal distribution, Fresnel, and geometric terms, respectively. (θ i ,φ i ) and (θ 0 ,φ 0 ) represent the height and azimuth of the incident light and the eye direction in the local coordinates, respectively. From the formula of the above BRDF model, it can be seen that when θ′ 0 When fixed, the BRDF is radially symmetric, or depends only on θ′ i This property can simplify the integral in formula (4).

[0104] integral

[0105] In order to Integrate in the local coordinates of L by the following method i (θ i ,φ i ) is converted to L i (θ′ i ,φ′ i ):

[0106]

[0107] represents the (2l+1)-dimensional representation of the rotation group SO(3). Inserting formulas (6) and (7) into formula (4), we can obtain the following formula:

[0108]

[0109] According to the radial symmetry property and the following formula:

[0110]

[0111] Where N lm′ is the normalization factor, P lmm is the associated Legendre function,

[0112]

[0113] If m′=0, then δ 0m′ =1, otherwise δ 0m′ =0. Then formula (8) can be simplified to

[0114]

[0115] ∧ lis a constant that can be calculated analytically, Θ l is a relatively complex integral and is difficult to solve in closed form. Therefore, in the Taylor series cosθ′ 0 Expand

[0116]

[0117] because And cosθ′ 0 <1, the higher-order terms can be ignored. We find that a 5th-degree polynomial is sufficient to solve our problem. Therefore, Θ l It can be approximated as a and cosθ′ 0 5th degree polynomial Since the camera's field of view (fov) is available, and cosθ′ 0 can be determined by the position of the pixel in the image, so Can be written as in, is the pixel position. The image is rendered with express.

[0118] To do this, the rendering process uses only screen space attributes. and and can be expressed as a low-cost linear combination of polynomials of these properties, and this combination is differentiable:

[0119]

[0120] In this embodiment, a small number of parameters can be used for rendering and prediction without losing much accuracy, thereby reducing the prediction and rendering costs.

[0121] In one embodiment, the loss of the neural network is calculated based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image, as described below.

[0122] A rendering loss is calculated based on the rendered image and the training image. An SH loss is calculated based on the predicted SH coefficients and the ground truth SH coefficients. An object normal map loss is calculated based on the predicted normal map of the object and the ground truth normal map of the object. A plane normal map loss is calculated based on the predicted normal map of the plane and the ground truth normal map of the plane. A reflectivity loss is calculated based on the predicted reflectivity and the ground truth reflectivity. A weighted sum of the rendering loss, the SH loss, the object normal map loss, the plane normal map loss, and the reflectivity loss is calculated as the loss.

[0123] In one embodiment, rather than requiring a large amount of data for modeling the intrinsic mapping from image to brightness, labeled training data is used to supervise the intermediate components to make the estimated brightness more reliable.

[0124] The ground truth SH coefficients and the ground truth normal map of the object can be extracted from the first dataset, so that the predicted SH coefficients and the predicted normal map of the object can be supervised. In addition, ARCore can also provide the ground truth normal map of the plane, so as to supervise the predicted plane normal map.

[0125] Rendering loss L r

[0126] L r The input image I representing the foreground and the rendered image of the foreground The pixel difference between 1 .

[0127]

[0128] Use l 1 The norm as a loss function helps to be robust to outliers, such as in Self-shadowing or extreme mirror reflections in I are ignored.

[0129] SH Loss L S

[0130] L S represents the mean square error (MSE) loss of the SH coefficient and is defined as:

[0131]

[0132] Represents the ground truth SH coefficient of the c-th color channel (in RGB). Represents the predicted SH coefficient of the c-th color channel (in RGB).

[0133] Object normal map loss L N

[0134] L N The measure is the ground truth normal map N of the object gt Predicted normal map with the extracted foreground object The pixel difference between 2 .

[0135]

[0136] Planar Normal Map Loss

[0137] for For example, the mandatory extension area (MA -M) have the same P The same normal map is not feasible. This is because the extracted foreground may contain some other non-planar objects in addition to the target object and the plane where the target object is located. One observation is that (M A Most of the pixels in (M) will lie on one plane. Therefore, it is reasonable to require that A -M) most but not all pixels are close to N P . The definition is:

[0138]

[0139] represents a subset of {·}, which is the smallest element in the first η% of {·}. In this embodiment, the closest N P The first η% pixels of 2 Minimize the distance to N P For example, η can be set to 80 based on experience.

[0140] Reflectivity loss L a

[0141] L a It is based on the similarity of color and intensity between pixels. The term is inspired by the multi-scale shading smoothing property. a Defined as the weighted L on neighboring pixels 2 Term, with weight as negative gradient amplitude:

[0142]

[0143] nb(i) represents the 8-connected neighborhood around pixel I, is the gradient of image I.

[0144] To this end, by combining the above five items, the final loss function can be defined as:

[0145]

[0146] In order to reduce the overfitting effect during training, additional regularization terms can be added to the above final loss function to further optimize the estimated SH coefficient using statistical regularization constraints. For example, the value of the weighting coefficient in the above final loss function can be λ r =1.92,λ a =0.074,λ s =2.14,λ n =1.01, andλ REG =2.9×10 -5Utilizing these losses during training, accurate SH brightness, normal maps, and reflectance can be produced with the help of the natural network.

[0147] Fig.10 The normal map estimation using two existing estimation methods and the method in this disclosure is compared. Fig.10 As shown, in both cases, the method in the present disclosure is significantly better. The two existing estimation methods are prone to over-smoothing the normal map due to manual regularization. In contrast, the method in the present disclosure can restore high-frequency details of the surface normal map even in the mirror and shadow areas.

[0148] Fig.11 is a schematic diagram showing an example of rendering a virtual object in a real scene. Fig.11 As shown, the left side is an image captured by the device, and the right side is an image synthesized by rendering the virtual object with the estimated brightness. The top row is a brighter scene, while the bottom row is a darker scene. Through the embodiments in the present application, the brightness conditions can be accurately estimated.

[0149] Fig.12 is a schematic diagram of a method for training a neural network according to an embodiment. Fig.12 As shown, the method for training a neural network includes the following.

[0150] In block 1202 , a training image is obtained from a first dataset, wherein the first dataset includes a plurality of images and ground truth parameters corresponding to each image, and the training image is any image in the first dataset.

[0151] Each image contains an object and the plane where the object is located. The ground truth parameters corresponding to each image include the ground truth normal map of the object, the ground truth normal map of the plane, the ground truth SH coefficient, the ground truth reflectivity and the ground truth roughness.

[0152] At block 1204, the foreground of the training image is extracted.

[0153] At block 1206, the extracted foreground of the training image is input into a neural network to obtain prediction parameters.

[0154] The prediction parameters include a predicted normal map of an object in the training image, a predicted normal map of a plane in the training image, a predicted SH coefficient, a predicted reflectivity, and a predicted roughness.

[0155] At block 1208, the image is rendered using the predicted parameters.

[0156] At block 1210 , a loss of the neural network is calculated based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image.

[0157] At block 1212, a neural network is trained based on the loss.

[0158] In one embodiment, the loss of the neural network is calculated based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image, as described below. The rendering loss is calculated based on the rendered image and the training image. The SH loss is calculated based on the predicted SH coefficient and the ground truth SH coefficient. The object normal map loss is calculated based on the predicted normal map of the object and the ground truth normal map of the object. The plane normal map loss is calculated based on the predicted normal map of the plane and the ground truth normal map of the plane. The reflectivity loss is calculated based on the predicted reflectivity and the ground truth reflectivity. The weighted sum of the rendering loss, the SH loss, the object normal map loss, the plane normal map loss, and the reflectivity loss is calculated as the loss.

[0159] In one embodiment, the method further includes the following.

[0160] Each image containing an object and a plane on which the object is located is selected from a second data set, wherein the second data set includes a plurality of panoramic images and a corresponding ground truth HDR brightness map in each panoramic image. For each selected image: a ground truth normal map of the object and a ground truth normal map of the plane are calculated; a ground truth SH coefficient corresponding to the image is extracted by applying spherical convolution to the ground truth HDR brightness map; and a ground truth reflectivity and a ground truth roughness corresponding to the image are calculated. A first data set is constructed based on the selected images and the ground truth parameters corresponding to each selected image, wherein the ground truth parameters corresponding to each selected image include the ground truth normal map of the object, the ground truth normal map of the plane, the ground truth SH coefficient, the ground truth reflectivity, and the ground truth roughness.

[0161] The above operations may refer to the detailed description of the network training operation in a method for rendering a virtual object based on brightness estimation, which will not be described here.

[0162] The above mainly introduces the scheme of the embodiment of the present invention from the perspective of the execution process on the method side. It can be understood that in order to realize the above functions, the mobile terminal includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0163] The embodiment of the present invention can divide the functional units of the device for rendering virtual objects based on brightness estimation and the device for training neural networks according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0164] Fig.13 is a schematic diagram of the structure of an apparatus for rendering a virtual object based on brightness estimation according to an embodiment. Figure 3 As shown, the apparatus for rendering a virtual object based on brightness estimation includes a shooting unit 131 , an extracting unit 132 , an estimating unit 133 and a rendering unit 134 .

[0165] The photographing unit 131 is configured to photograph an image in which at least one object is located on at least one plane.

[0166] The extraction unit 132 is configured to extract the foreground of the image.

[0167] The estimation unit 133 is configured to estimate the brightness corresponding to the extracted foreground.

[0168] The rendering unit 134 is configured to render the virtual object with the estimated brightness.

[0169] It can be seen that in the embodiment of the present application, an image in a real scene is captured in real time, at least one object in the image is located on at least one plane, and then the foreground of the image is extracted, and then the brightness corresponding to the extracted foreground is estimated, and finally the virtual object is rendered in the real scene with the estimated brightness. Therefore, through the input of the trained neural network and the image captured in the real scene containing one or more objects placed on the plane area, the brightness conditions of the real world can be estimated in real time, and the virtual object can be rendered in the real scene with the estimated brightness, thereby improving the rendering quality and realism of the virtual object rendered in the real scene.

[0170] In one embodiment, the estimation unit 133 is configured to: input the extracted foreground into a neural network; obtain predicted spherical harmonics (SH) coefficients output by the neural network; and determine the brightness corresponding to the image based on the predicted SH coefficients.

[0171] In one embodiment, the extraction unit 132 is configured to: detect the presence of at least one preset object in the image; select an object located at the center of the image as a target object from the at least one preset object; and determine and extract the foreground of the image based on the target object and the plane where the target object is located.

[0172] In one embodiment, in selecting an object located at the center of an image from at least one preset object as a target object, the extraction unit 132 is configured to: select an object at least partially approximately located at the center of the image from at least one preset object as the target object.

[0173] In one embodiment, in determining the foreground of an image based on a target object and a plane where the target object is located, the estimation unit 133 is configured to: determine a bounding box based on the target object; divide the bounding box into an upper bonding box and a lower bonding box; expand the lower bounding box at a magnification factor to include a portion of the plane where the target object is located; and determine a portion of the image framed by the upper bounding box and the expanded lower bounding box as the foreground of the image.

[0174] In one embodiment, the apparatus further comprises a training unit 135 .

[0175] The training unit 135 is configured to: train a neural network with a first data set, wherein the first data set includes multiple images and ground truth parameters corresponding to each image, each image contains an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, a ground truth SH coefficient, a ground truth reflectivity, and a ground truth roughness.

[0176] As an embodiment, the apparatus further includes a construction unit 136 .

[0177] The construction unit 136 is configured to: select multiple images from the second data set, each of the multiple images contains an image of an object and a plane where the object is located, wherein the second data set includes multiple panoramic images and a ground truth HDR brightness map corresponding to each panoramic image; for each selected image: calculate the ground truth normal map of the object and the ground truth normal map of the plane, extract the ground truth SH coefficient corresponding to the image by performing spherical convolution on the ground truth HDR brightness, and calculate the ground truth reflectivity and ground truth roughness corresponding to the image; construct a first data set based on the selected image and the ground truth parameters corresponding to the selected image, wherein the ground truth parameters corresponding to each selected image include the ground truth normal map of the object, the ground truth normal map of the plane, the ground truth SH coefficient, the ground truth reflectivity and the ground truth roughness.

[0178] In one embodiment, the training unit 136 is configured to: obtain a training image from a first data set, wherein the training image is any image in the first data set; extract the foreground of the training image; input the extracted foreground of the training image into a neural network to obtain prediction parameters, the prediction parameters including a predicted normal map of an object in the training image, a predicted normal map of a plane in the training image, a predicted SH coefficient, a predicted reflectivity, and a predicted roughness; render the image based on the predicted parameters; calculate the loss of the neural network based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image; and train the neural network based on the loss.

[0179] In one embodiment, in terms of calculating the loss of the neural network based on the predicted parameters, the rendered image, the training image and the ground truth parameters corresponding to the training image, the training unit 136 is configured to: calculate the rendering loss based on the rendered image and the training image; calculate the SH loss based on the predicted SH coefficient and the ground truth SH coefficient; calculate the object normal map loss based on the predicted normal map of the object and the ground truth normal map of the object; calculate the plane normal map loss based on the predicted normal map of the plane and the ground truth normal map of the plane; calculate the reflectivity loss based on the predicted reflectivity and the ground truth reflectivity; calculate the weighted sum of the rendering loss, the SH loss, the object normal map loss, the plane normal map loss and the reflectivity loss as the loss.

[0180] Fig.14 is a schematic diagram of the structure of an apparatus for training a neural network according to an embodiment. Fig.14 As shown, the apparatus for training a neural network includes an acquisition unit 141 , an extraction unit 142 , an input unit 143 , a rendering unit 144 , a calculation unit 145 and a training unit 146 .

[0181] The acquisition unit 141 is used to acquire a training image from a first data set, wherein the first data set includes images and ground truth parameters corresponding to each image, and the training image is any image in the first data set.

[0182] Each image contains an object and the plane where the object is located. The ground truth parameters corresponding to each image include the ground truth normal map of the object, the ground truth normal map of the plane, the ground truth SH coefficient, the ground truth reflectivity and the ground truth roughness.

[0183] The extraction unit 142 is configured to extract the foreground of the training image.

[0184] The input unit 143 is configured to input the extracted foreground of the training image into the neural network to obtain prediction parameters.

[0185] The prediction parameters include a predicted normal map of an object in the training image, a predicted normal map of a plane in the training image, a predicted SH coefficient, a predicted reflectivity, and a predicted roughness.

[0186] The rendering unit 144 is configured to render an image based on the predicted parameters;

[0187] The computing unit 145 is configured to compute the loss of the neural network based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image.

[0188] The training unit 146 is configured to train the neural network based on the loss.

[0189] In one embodiment, the computing unit 145 is configured to calculate a rendering loss based on the rendered image and the training image; calculate an SH loss based on the predicted SH coefficient and the ground truth SH coefficient; calculate an object normal map loss based on the predicted normal map of the object and the ground truth normal map of the object; calculate a plane normal map loss based on the predicted normal map of the plane and the ground truth normal map of the plane; calculate a reflectivity loss based on the predicted reflectivity and the ground truth reflectivity; and calculate a weighted sum of the rendering loss, the SH loss, the object normal map loss, the plane normal map loss, and the reflectivity loss as a loss.

[0190] As an embodiment, the device also includes a construction unit 147. The construction unit is configured to: select multiple images from the second data set, each of the multiple images contains an image of the object and the plane where the object is located, wherein the second data set includes multiple panoramic images and a ground truth HDR brightness map corresponding to each panoramic image; for each selected image: calculate the ground truth normal map of the object and the ground truth normal map of the plane, apply spherical convolution to the ground truth HDR brightness map to extract the ground truth SH coefficient corresponding to the image, and calculate the ground truth reflectivity and ground truth roughness corresponding to the image; construct a first data set based on the selected image and the ground truth parameters corresponding to the selected image, wherein the ground truth parameters corresponding to each selected image include the ground truth normal map of the object, the ground truth normal map of the plane, the ground truth SH coefficient, the ground truth reflectivity, and the ground truth roughness.

[0191] Fig.15 is a schematic diagram of the structure of a terminal device according to an embodiment. Fig.15 As shown, the terminal device 150 includes a processor 151, a memory 152, a communication interface 153, and one or more programs 154 stored in the memory 152 and executed by the processor 151. The one or more programs 154 include instructions for performing the following operations.

[0192] An image is captured, wherein at least one object in the image is located on at least one plane. A foreground of the image is extracted. A brightness corresponding to the extracted foreground is estimated. A virtual object is rendered based on the estimated brightness.

[0193] In one embodiment, in terms of estimating the brightness corresponding to the extracted foreground, the one or more programs 154 include instructions for performing the following operations: Inputting the extracted foreground into a neural network Obtaining predicted values ​​of spherical harmonic (SH) coefficients output by the neural network Determining the brightness corresponding to the image based on the predicted SH coefficients

[0194] In one embodiment, in extracting the foreground of an image, the one or more programs 154 include instructions for performing the following operations: Detecting the presence of at least one preset object in the image. Selecting an object located at the center of the image as a target object from the at least one preset object. Determining and extracting the foreground of the image based on the target object and the plane where the target object is located.

[0195] In one embodiment, in selecting an object located at the center of the image from at least one preset object as the target object, the one or more programs 154 include instructions for performing the following operations: Selecting an object located at least partially approximately at the center of the image from at least one preset object as the target object.

[0196] In one embodiment, in determining the foreground of an image based on a target object and a plane where the target object is located, the one or more programs 154 include instructions for performing the following operations: determining a bounding box according to the target object; dividing the bounding box into an upper bounding box and a lower bounding box; extending the lower bounding box to include a portion of the plane where the target object is located; and determining that the portion of the image surrounded by the upper bounding box and the extended lower bounding box is the foreground of the image.

[0197] In one embodiment, the one or more programs 154 further include instructions for performing the following operations: training a neural network with a first data set, wherein the first data set includes a plurality of images and ground truth parameters corresponding to each image, each image includes an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, a ground truth SH coefficient, a ground truth reflectivity, and a ground truth roughness.

[0198] In one embodiment, one or more programs 154 further include instructions for performing the following operations. A plurality of images are selected from a second data set, each of the plurality of images containing an object and a plane on which the object is located, wherein the second data set includes a plurality of panoramic images and a ground truth HDR brightness map corresponding to each panoramic image. For each selected image: a ground truth normal map of the object and a ground truth normal map of the plane are calculated; a spherical convolution is applied to the ground truth HDR brightness map to extract the ground truth spherical harmonic coefficients corresponding to the image; and a ground truth reflectivity and a ground truth roughness of the image are calculated. A first data set is constructed based on the selected image and the ground truth parameters corresponding to each selected image, wherein the ground truth parameters corresponding to each selected image include a ground truth normal map of the object, a ground truth normal map of the plane, ground truth spherical harmonic coefficients, ground truth reflectivity, and ground truth roughness.

[0199] In one embodiment, in training a neural network with the first data set, one or more programs 154 include instructions for performing the following operations. Obtain a training image from the first data set, the training image being any image in the first data set. Extract a foreground of the training image. Input the extracted foreground of the training image into the neural network to obtain prediction parameters, the prediction parameters including a predicted normal map of an object in the training image, a predicted normal map of a plane in the training image, a predicted spherical harmonic coefficient, a predicted reflectivity, and a predicted roughness. Render an image based on the prediction parameters. Calculate a loss of the neural network based on the prediction parameters, the rendered image, the training image, and ground truth parameters corresponding to the training image. Train the neural network based on the loss.

[0200] In one embodiment, in terms of calculating the loss of a neural network based on prediction parameters, a rendered image, a training image, and ground truth parameters corresponding to the training image, one or more programs 154 include instructions for performing the following operations. Calculate a rendering loss based on the rendered image and the training image. Calculate an SH loss based on the predicted SH coefficients and the ground truth SH coefficients. Calculate an object normal map loss based on the predicted normal map of the object and the ground truth normal map of the object. Calculate a plane normal map loss based on the predicted normal map of the plane and the ground truth normal map of the plane. Calculate a reflectivity loss based on the predicted reflectivity and the ground truth reflectivity. Calculate a weighted sum of the rendering loss, the SH loss, the object normal map loss, the plane normal map loss, and the reflectivity loss as a loss.

[0201] Fig.16 is a schematic diagram of the structure of a terminal device according to another embodiment. Fig.16As shown, the terminal device 160 includes a processor 161, a memory 162, a communication interface 163, and one or more programs 164 stored in the memory 162 and executed by the processor 161. The one or more programs 164 include instructions for performing the following operations.

[0202] A training image is obtained from a first data set, the first data set comprising a plurality of images and ground truth parameters corresponding to each image, the training image being any image included in the first data set. A foreground of the training image is extracted. The extracted foreground of the training image is input into a neural network to obtain prediction parameters. An image is rendered using the prediction parameters. A loss of the neural network is calculated based on the prediction parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image. The neural network is trained based on the loss.

[0203] In one embodiment, each image contains an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, a ground truth SH coefficient, a ground truth reflectivity, and a ground truth roughness.

[0204] In one embodiment, the predicted parameters include predicted normal maps of objects in the training images, predicted normal maps of planes in the training images, predicted SH coefficients, predicted reflectivity, and predicted roughness. The image is rendered based on the predicted parameters.

[0205] In one embodiment, in terms of calculating the loss of the neural network based on the predicted parameters, the rendered image, the training image, and the ground truth parameters corresponding to the training image, one or more programs 164 include instructions for performing the following operations. Calculate a rendering loss based on the rendered image and the training image. Calculate an SH loss based on the predicted SH coefficients and the ground truth SH coefficients. Calculate an object normal map loss based on the predicted normal map of the object and the ground truth normal map of the object. Calculate a plane normal map loss based on the predicted normal map of the plane and the ground truth normal map of the plane. Calculate a reflectivity loss based on the predicted reflectivity and the ground truth reflectivity. Calculate a weighted sum of the rendering loss, the SH loss, the object normal map loss, the plane normal map loss, and the reflectivity loss as a loss.

[0206] In one embodiment, one or more programs 164 further include instructions for performing the following operations. A plurality of images are selected from a second data set, each of the plurality of images containing an object and a plane on which the object is located, wherein the second data set includes a plurality of panoramic images and a ground truth HDR brightness map corresponding to each panoramic image. For each selected image: a ground truth normal map of the object and a ground truth normal map of the plane are calculated; a spherical convolution is applied to the ground truth HDR brightness map to extract the ground truth spherical harmonic coefficients corresponding to the image; and a ground truth reflectivity and a ground truth roughness of the image are calculated. A first data set is constructed based on the selected image and the ground truth parameters corresponding to each selected image, wherein the ground truth parameters corresponding to each selected image include a ground truth normal map of the object, a ground truth normal map of the plane, ground truth spherical harmonic coefficients, ground truth reflectivity, and ground truth roughness.

[0207] The present application also provides a non-transitory computer storage medium. The non-transitory computer storage medium is configured to store a program, which, when executed, is operable to perform some or all operations of the method for rendering a virtual object based on brightness estimation as described in the above method embodiment, or to perform some or all operations of the method for training a neural network.

[0208] The embodiment of the present application also provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to perform some or all operations of the method for rendering a virtual object based on brightness estimation as described in the above method embodiment, or to perform some or all operations of the method for training a neural network.

[0209] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0210] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the unit, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method of each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0215] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0216] The embodiments of the present application are described in detail above, and specific examples are used here to describe the principles and implementations of the present disclosure. The description of the above embodiments is only used to help understand the methods and core ideas of the present disclosure. At the same time, those skilled in the art can modify the specific implementations and application scopes according to the ideas of the present disclosure. In a word, the contents of the specification should not be interpreted as limiting the present disclosure.

Claims

1. A method for rendering a virtual object based on brightness estimation, comprising: capturing an image, wherein at least one object in the image is located on at least one plane; extracting a foreground of the image; Estimating the brightness corresponding to the extracted foreground, comprising: inputting the extracted foreground into a neural network to obtain the brightness corresponding to the foreground; and rendering a virtual object based on the estimated brightness; The method further comprises: The neural network is trained using a first data set, the first data set comprising a plurality of images and ground truth parameters corresponding to each image, wherein each image comprises an object and a plane on which the object is located, and the ground truth parameters corresponding to each image comprise a ground truth normal map of the object, a ground truth normal map of the plane, ground truth spherical harmonic coefficients, ground truth reflectivity, and ground truth roughness.

2. The method according to claim 1, wherein: Inputting the extracted foreground into a neural network to obtain the brightness corresponding to the foreground includes: Inputting the extracted foreground into a neural network; Obtaining predicted spherical harmonics (SH) coefficients output by the neural network; and Based on the predicted spherical harmonic coefficients, the brightness corresponding to the foreground is determined.

3. The method according to claim 1 or 2, wherein: Extracting the foreground of the image includes: detecting that at least one preset object exists in the image; Selecting an object located at the center of the image from the at least one preset object as a target object; and The foreground of the image is determined and extracted based on the target object and the plane where the target object is located.

4. The method according to claim 3, wherein: Determining the foreground of the image based on the target object and the plane where the target object is located includes: determining a bounding box according to the target object; dividing the bounding box into an upper bounding box and a lower bounding box; extending the lower bounding box to include a portion of the plane in which the target object lies; and An image portion surrounded by the upper bounding box and the extended lower bounding box is determined as a foreground of the image.

5. The method according to claim 1, further comprising: Selecting a plurality of images from a second data set, each of the plurality of images comprising an object and a plane where the object is located, wherein the second data set comprises a plurality of panoramic images and a ground truth HDR brightness map corresponding to each panoramic image; For each selected image: Compute the ground truth normal map of the object and the ground truth normal map of the plane; Extract the ground truth spherical harmonic coefficients corresponding to the image by applying spherical convolution to the ground truth HDR brightness map; Calculating the ground truth reflectance and the ground truth roughness of the image; and A first data set is constructed based on the selected multiple images and ground truth parameters corresponding to the selected multiple images, wherein the ground truth parameters corresponding to each selected image include a ground truth normal map of the object, a ground truth normal map of the plane, ground truth spherical harmonic coefficients, ground truth reflectivity, and ground truth roughness.

6. The method according to claim 5, wherein: Training the neural network with the first data set includes: Acquire a training image from the first data set, where the training image is any image in the first data set; extracting the foreground of the training image; Inputting the extracted foreground of the training image into a neural network to obtain prediction parameters, the prediction parameters including a predicted normal map of an object in the training image, a predicted normal map of a plane in the training image, a predicted spherical harmonic coefficient, a predicted reflectivity, and a predicted roughness; rendering an image using the prediction parameters; Calculating a loss of the neural network based on the prediction parameters, the rendered image, the training image, and ground truth parameters corresponding to the training image; and The neural network is trained based on the loss.

7. The method according to claim 6, wherein: Calculating the loss of the neural network based on the prediction parameters, the rendered image, the training image, and ground truth parameters corresponding to the training image includes: Calculating rendering loss based on the rendered image and the training image; Calculate spherical harmonic loss based on predicted spherical harmonic coefficients and ground truth spherical harmonic coefficients; Compute an object normal map loss based on the object's predicted normal map and the object's ground truth normal map; Compute a plane normal map loss based on the predicted normal map of the plane and the ground truth normal map of the plane; Calculating a roughness loss based on the predicted roughness and the ground truth roughness; and A weighted sum of the rendering loss, the spherical harmonics loss, the object normal map loss, the plane normal map loss, and the roughness loss is calculated as the loss.

8. A method for training a neural network, comprising: Acquire a training image from a first data set, where the first data set includes a plurality of images and a ground truth parameter corresponding to each image, and the training image is any image included in the first data set; extracting the foreground of the training image; Inputting the extracted foreground of the training image into a neural network to obtain prediction parameters; rendering an image using the prediction parameters; Calculating a loss of the neural network based on the prediction parameters, the rendered image, the training image, and ground truth parameters corresponding to the training image; as well as training a neural network based on the loss; Each image contains an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, a ground truth spherical harmonic coefficient, a ground truth reflectivity, and a ground truth roughness; the prediction parameters include a predicted normal map of the object in the training image, a predicted normal map of the plane in the training image, a predicted spherical harmonic coefficient, a predicted reflectivity, and a predicted roughness.

9. The method according to claim 8, wherein: Calculating the loss of the neural network based on the prediction parameters, the rendered image, the training image, and ground truth parameters corresponding to the training image includes: Calculating rendering loss based on the rendered image and the training image; Calculating spherical harmonic loss based on the predicted spherical harmonic coefficients and the ground truth spherical harmonic coefficients; computing an object normal map loss based on a predicted normal map of the object and a ground truth normal map of the object; Calculating a plane normal map loss based on a predicted normal map of the plane and a ground truth normal map of the plane; Calculating a roughness loss based on the predicted roughness and the ground truth roughness; and A weighted sum of the rendering loss, the spherical harmonics loss, the object normal map loss, the plane normal map loss, and the roughness loss is calculated as the loss.

10. The method according to claim 8, further comprising: Selecting a plurality of images from a second data set, each of the plurality of images comprising an object and a plane where the object is located, wherein the second data set comprises a plurality of panoramic images and a ground truth HDR brightness map corresponding to each panoramic image; For each selected image: Compute the ground truth normal map of the object and the ground truth normal map of the plane; extracting ground truth spherical harmonic coefficients corresponding to the image by applying spherical convolution to the ground truth HDR brightness map; Calculating the ground truth reflectance and the ground truth roughness of the image; and The first data set is constructed based on the selected multiple images and ground truth parameters corresponding to the selected multiple images, wherein the ground truth parameters corresponding to each selected image include a ground truth normal map of the object, a ground truth normal map of the plane, ground truth spherical harmonic coefficients, ground truth reflectivity, and ground truth roughness.

11. A device for rendering a virtual object based on brightness estimation, comprising: a photographing unit, configured to photograph an image, wherein at least one object in the image is located on at least one plane; An extraction unit, used for extracting the foreground of the image; An estimating unit, configured to estimate the brightness corresponding to the extracted foreground, comprising: inputting the extracted foreground into a neural network to obtain the brightness corresponding to the foreground; and a rendering unit, configured to render a virtual object based on the estimated brightness; A training unit is used to train the neural network with a first data set, wherein the first data set includes a plurality of images and ground truth parameters corresponding to each image, wherein each image includes an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, ground truth spherical harmonic coefficients, ground truth reflectivity, and ground truth roughness.

12. The device according to claim 11, wherein: The estimation unit is specifically used for: Inputting the extracted foreground into a neural network; Obtaining predicted spherical harmonics (SH) coefficients output by the neural network; and Based on the predicted spherical harmonic coefficients, the brightness corresponding to the foreground is determined.

13. The device according to claim 11 or 12, wherein: The extraction unit is used for: detecting that at least one preset object exists in the image; Selecting an object located at the center of the image from the at least one preset object as a target object; as well as The foreground of the image is determined based on the target object and the plane where the target object is located.

14. The device according to claim 13, wherein: In determining the foreground of the image based on the target object and the plane where the target object is located, the estimation unit is used to: determining a bounding box according to the target object; dividing the bounding box into an upper bounding box and a lower bounding box; extending the lower bounding box to include a portion of the plane where the target object is located; as well as An image portion surrounded by the upper bounding box and the extended lower bounding box is determined as a foreground of the image.

15. An apparatus for training a neural network, comprising: an acquisition unit, configured to acquire a training image from a first data set, wherein the first data set includes a plurality of images and a ground truth parameter corresponding to each image, and the training image is any image included in the first data set; An extraction unit, used for extracting the foreground of the training image; An input unit, used for inputting the extracted foreground of the training image into a neural network to obtain prediction parameters; A rendering unit, configured to render an image using the prediction parameters; a computing unit, configured to compute a loss of the neural network based on prediction parameters, a rendered image, the training image, and ground truth parameters corresponding to the training image; as well as A training unit, configured to train a neural network based on the loss; Each image contains an object and a plane where the object is located, and the ground truth parameters corresponding to each image include a ground truth normal map of the object, a ground truth normal map of the plane, a ground truth spherical harmonic coefficient, a ground truth reflectivity, and a ground truth roughness; the prediction parameters include a predicted normal map of the object in the training image, a predicted normal map of the plane in the training image, a predicted spherical harmonic coefficient, a predicted reflectivity, and a predicted roughness.

16. A terminal device comprising a processor and a memory for storing one or more programs, wherein: The one or more programs are used to be executed by the processor, and the one or more programs include programs for executing the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 10.

17. A non-transitory computer-readable storage medium for storing a computer program for electronic data exchange, which, when executed, causes a computer to perform the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 10.

Images