Method for estimating light for augmented reality and electronic device thereof

Abstract

A method for operating an electronic device for presenting an augmented reality object is provided. The method includes: acquiring at least one first image of an environment in which an augmented reality virtual object is to be displayed; acquiring an environment texture based on the acquired at least one first image; identifying at least one light source in at least one second image by identifying at least one cluster including pixels having a brightness value greater than a threshold; determining the position of at least one light source based on the position of the pixel within the at least one cluster; presenting an augmented reality virtual object based on the acquired environment texture and the determined position of at least one light source, and displaying the presented augmented reality virtual object on a display of the electronic device.

Description

Technical Field

[0001] The present disclosure relates to presenting augmented reality objects. More specifically, the present disclosure relates to methods and devices for presenting augmented reality objects. Background Art

[0002] Augmented reality refers to the process by which computer-generated objects are superimposed on a user's view of the real world, providing a composite view through which the user can see and interact with the objects. In order to provide a more realistic composite view, it is necessary to closely match the appearance of the rendered objects to the surrounding real-world environment. To this end, computer vision and computer graphics algorithms have been developed that attempt to estimate the lighting conditions of the real-world environment in which the rendered objects will be placed, in order to produce more convincing results. Existing light estimation techniques can be categorized as approximation techniques and extrapolation techniques.

[0003] Approximate techniques are those that use a chosen camera system to record the environment, such as a red-green-blue (RGB) or RGB-depth (RGB-D) camera equipped with a rectilinear or fisheye lens, and approximate the lighting of the environment based solely on this information. These techniques tend to rely on certain assumptions, such as that diffuse light varies slowly in space.

[0004] Extrapolation techniques tend to infer the light in a particular environment from one or more images using machine learning algorithms that have been trained to estimate the light under multiple conditions. The final results of these techniques depend on the quality of the training data, and can often achieve results that are superior to approximate techniques. However, the disadvantages of extrapolation techniques include the extensive offline processing required to train the model, and a higher runtime cost compared to most approximate techniques.

[0005] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the invention

[0006] Solution to the problem

[0007] An aspect of the present disclosure is to address at least the above problems and / or disadvantages and to provide at least the advantages described below. Therefore, one aspect of the present invention is to provide a method and apparatus for augmented reality presentation.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0009] According to some embodiments of the present disclosure, a method for operating an electronic device for presenting an augmented reality object is provided. The method includes: acquiring at least one first image of an environment in which an augmented reality virtual object is to be displayed; acquiring an environment texture based on the acquired at least one first image; identifying at least one light source in at least one second image by identifying at least one cluster including pixels having a brightness value greater than a threshold; determining the position of at least one light source based on the position of the pixel in at least one cluster; presenting an augmented reality virtual object based on the acquired environment texture and the determined position of at least one light source, and displaying the presented augmented reality virtual object on a display of the electronic device.

[0010] According to some embodiments of the present disclosure, acquiring at least one first image of the environment includes acquiring a first straight line image and a second straight line image, and acquiring the environment texture includes deforming the first straight line image and the second straight line image according to a spherical projection.

[0011] According to some embodiments of the present disclosure, acquiring at least one first image of the environment includes acquiring an image to be displayed on a display of an electronic device corresponding to a head mounted display (HMD) device, and acquiring the environment texture includes deforming the acquired image.

[0012] According to some embodiments of the present disclosure, identifying at least one light source includes: setting a first value to the first pixel based on identifying that the brightness value of the first pixel in at least one second image is greater than a threshold; and setting a second value to the second pixel based on identifying that the brightness value of the second pixel in at least one second image is less than a threshold.

[0013] According to some embodiments of the present disclosure, determining the position of at least one light source includes identifying a geometric center point of pixels in each of at least one cluster, and setting the geometric center point as the position of each of the at least one light source.

[0014] According to some embodiments of the present disclosure, the method also includes determining the relative intensities of two or more light sources among at least one light source based on the number of pixels in each of at least one cluster, and one light source corresponding to a cluster with more pixels is determined to have a higher intensity than another light source corresponding to a cluster with fewer pixels.

[0015] According to some embodiments of the present disclosure, the at least one second image includes an environment texture acquired based on the at least one first image.

[0016] According to some embodiments of the present disclosure, the method further comprises acquiring at least one second image based on a lower exposure setting than the at least one first image.

[0017] According to some embodiments of the present disclosure, identifying at least one light source includes: identifying the position of a light source that exists outside the acquired image among the light sources in the environment based on at least one of the direction of the shadow of the object in the acquired image, the size of the shadow, or the brightness values ​​of pixels associated with the object other than the at least one light source.

[0018] According to some embodiments of the present disclosure, an electronic device for presenting an augmented reality object is provided. The electronic device includes a display, an image sensor, at least one processor connected to the display and the image sensor, and a memory connected to the at least one processor. The memory includes instructions that can be executed by at least one processor and configured to cause at least one processor to use the image sensor to acquire at least one first image of an environment in which an augmented reality virtual object is to be displayed, identify at least one light source in at least one second image by identifying at least one cluster including pixels having a brightness value greater than a threshold, determine the position of at least one light source based on the position of the pixel in at least one cluster, present the augmented reality virtual object based on the acquired environment texture and the determined position of at least one light source, and display the presented augmented reality virtual object on the display.

[0019] According to some embodiments of the present disclosure, in order to obtain at least one first image of the environment, at least one processor is further configured to obtain a first straight line image and a second straight line image by using an image sensor. In addition, in order to obtain the texture of the environment, at least one processor is further configured to deform the first straight line image and the second straight line image according to a spherical projection.

[0020] According to some embodiments of the present disclosure, in order to obtain at least one first image of the environment, at least one processor is further configured to obtain an image to be displayed on a display of an electronic device corresponding to a head mounted display (HMD) device by using an image sensor. In addition, in order to obtain the texture of the environment, at least one processor is further configured to deform the obtained image.

[0021] According to some embodiments of the present disclosure, in order to identify at least one light source, at least one processor is also configured to set a first value to a first pixel based on identifying that a brightness value of the first pixel in at least one second image is greater than a threshold, and to set a second value to a second pixel based on identifying that a brightness value of the second pixel in at least one second image is less than a threshold.

[0022] According to some embodiments of the present disclosure, in order to determine the position of at least one light source, at least one processor is further configured to identify the geometric center point of pixels in each cluster in at least one cluster, and set the geometric center point as the position of each light source in the at least one light source.

[0023] According to some embodiments of the present disclosure, at least one processor is also configured to determine the relative intensities of two or more light sources among at least one light source based on the number of pixels in each of at least one cluster, and one light source corresponding to a cluster with more pixels is determined to have a higher intensity than another light source corresponding to a cluster with fewer pixels.

[0024] According to some embodiments of the present disclosure, the at least one second image includes an environment texture acquired based on the at least one first image.

[0025] According to some embodiments of the present disclosure, the at least one processor is further configured to acquire at least one second image based on a lower exposure setting than the at least one first image.

[0026] According to some embodiments of the present disclosure, in order to identify at least one light source, at least one processor is further configured to identify the position of a light source that exists outside the acquired image among the light sources in the environment based on at least one of the direction of the shadow of the object in the acquired image, the size of the shadow, or the brightness values ​​of pixels associated with the object other than the at least one light source.

[0027] According to some embodiments of the present disclosure, a computer program product is provided. The computer program product includes computer program instructions, which perform the following method when executed, the method comprising: acquiring at least one first image of an environment in which an augmented reality virtual object is to be displayed; acquiring an environment texture based on the acquired at least one first image; identifying at least one light source in at least one second image by identifying at least one cluster including pixels having a brightness value greater than a threshold; determining the position of at least one light source based on the position of the pixel in at least one cluster, presenting an augmented reality virtual object based on the acquired environment texture and the determined position of at least one light source, and displaying the presented augmented reality virtual object on a display of an electronic device.

[0028] According to some embodiments of the present disclosure, at least one non-transitory computer-readable storage medium arranged to store a computer program product is provided. The computer program product includes computer program instructions, which perform the following method when executed, the method comprising: acquiring at least one first image of an environment in which an augmented reality virtual object is to be displayed; acquiring an environment texture based on the acquired at least one first image; identifying at least one light source in at least one second image by identifying at least one cluster including pixels having a brightness value greater than a threshold; determining the position of at least one light source based on the position of the pixel in at least one cluster, presenting an augmented reality virtual object based on the acquired environment texture and the determined position of at least one light source, and displaying the presented augmented reality virtual object on a display of an electronic device.

[0029] Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 An equivalent two-dimensional (2D) representation of a spherical field of view according to an embodiment of the present disclosure is shown;

[0032] Figure 2 is a flowchart illustrating a method of presenting an augmented reality object according to an embodiment of the present disclosure;

[0033] Figure 3 is a flowchart showing a method of combining images to obtain environment texture according to an embodiment of the present disclosure;

[0034] Figure 4 schematically illustrates a first captured image captured according to an embodiment of the present disclosure;

[0035] Figure 5 schematically illustrates a second captured image captured according to an embodiment of the present disclosure;

[0036] Figure 6 Schematically illustrates an environment texture acquired by combining a first image and a second image according to an embodiment of the present disclosure;

[0037] Figure 7 is a flow chart illustrating a method of determining the position of one or more light sources according to an embodiment of the present disclosure;

[0038] Figure 8 Schematically illustrates a device for presenting an augmented reality object according to an embodiment of the present disclosure;

[0039] Fig. 9 An image showing an environment including a plurality of light sources according to an embodiment of the present disclosure;

[0040] Fig.10 An image showing an environment including a plurality of light sources after blurring according to an embodiment of the present disclosure; and

[0041] Fig.11 An image of an environment including multiple light sources after thresholding is shown in accordance with an embodiment of the present disclosure.

[0042] Throughout the drawings, like reference numerals will be understood to refer to like parts, components and structures. DETAILED DESCRIPTION

[0043] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes specific details to assist understanding, but these are considered to be exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0044] The terms and words used in the following specification and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for limiting the present disclosure as defined by the appended claims and their equivalents.

[0045] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0046] Figure 1 An equivalent two-dimensional (2D) representation of a spherical field of view according to an embodiment of the present disclosure is shown.

[0047] Reference Figure 1 , an augmented reality object can be rendered based on an environment texture that represents the environment surrounding the object. The environment texture can represent a spherical field of view centered on the user's current location. The environment texture can be stored in various formats, such as Figure 1 , which shows an equivalent 2D representation of a spherical field of view according to an embodiment of the present disclosure. The spherical field of view (also referred to as a 360-degree field of view) can be divided into a front field of view, a rear field of view, a top field of view, a bottom field of view, a left field of view, and a right field of view.

[0048] In the unit sphere representation, pixels of the image data are indexed in a three-dimensional coordinate system that defines the location of each pixel on the surface of the virtual sphere 110. Image data for a spherical field of view may also be represented in various 2D formats, such as a cubic map projection 120 and an equirectangular projection 130. In the cubic map projection 120, the pixel data for each face of the virtual cube may be stored as a 2D image spanning a 90°×90° field of view. In the equirectangular projection 130, the pixel data is stored as a single 2D image spanning a 360°×180° field of view. Figure 1, the labels “top,” “bottom,” “front,” “back,” “left,” and “right” indicate corresponding areas of pixel data in each equivalent projection.

[0049] Figure 2 is a flowchart illustrating a method of presenting an augmented reality object according to an embodiment of the present disclosure.

[0050] Reference Figure 2 , at operation S201, the electronic device may capture a first image and a second image of an environment in which an augmented reality virtual object is to be displayed. In an embodiment of the present disclosure, the first image may be captured using a front camera of the electronic device, and the second image may be captured using a rear camera of the electronic device. For example, since many modern electronic devices (such as smart phones or tablet computers) include front cameras and rear cameras, the first image and the second image may be acquired using common hardware in electronic devices. However, any image capture device may be used to capture the first image and / or the second image.

[0051] Next, at operation S202, the electronic device may combine the first image and the second image to obtain an environment texture. In an embodiment of the present disclosure, when the first and second images are captured from both the front camera and the rear camera, the images may be combined to obtain a relatively complete environment texture. For example, since most of the surrounding environment will be visible to the front camera or the rear camera (e.g., within the field of view), the first and second images may capture the surrounding environment so that the environment around the electronic device may be represented in the acquired environment texture.

[0052] At operation S203, the electronic device may determine the position of one or more light sources associated with the environment surrounding the electronic device. For example, the position of the light source within the environment may be determined by analyzing an image of the environment to detect or identify the one or more light sources. According to an embodiment of the present disclosure, the position of the one or more light sources may be determined by analyzing the same image (e.g., the first image and the second image) used to obtain the texture of the environment or by analyzing other images of the environment.

[0053] In operation S204, the electronic device may present an augmented reality virtual object based on the acquired environmental texture and the determined position of one or more light sources. By simultaneously considering the surrounding environmental texture and the position of one or more light sources, the object may be presented in a more convincing manner so that the appearance of the presented object closely matches the actual lighting conditions in the current environment surrounding the electronic device. In an embodiment of the present disclosure, the presented augmented reality virtual object may then be combined with an image of the environment surrounding the electronic device to obtain a composite image including the presented object. In some embodiments of the present disclosure, the electronic device may also be configured to display the acquired composite image.

[0054] According to an embodiment of the present disclosure, an electronic device may be operated as a head mounted display (HMD) device. The electronic device may acquire an image of an area in an environment displayed on a head-up display (HUD). The electronic device may acquire an environmental texture from the acquired image displayed on the HUD. The electronic device may determine the position of one or more light sources based on the acquired image. For example, the position and / or main direction of the light source may be determined based on parameters related to one or more light sources identified as being present in the acquired image, such as a brightness value associated with each light source, the size of each light source, etc. In addition, the position of the light source that is not directly present in the acquired image among the light sources present in the environment is determined based on the following parameters. For example, by identifying the direction of the shadow of an object other than the light source in the acquired image, the size of the shadow, and / or the brightness of the object, the position, size, and / or brightness of one or more light sources that are not present in the acquired image may be identified. In addition, while identifying the above parameters, the color of each light source that is not present in the acquired image and in the acquired image may be identified together. The electronic device may present an augmented reality virtual object based on the acquired environmental texture and one or more light sources identified based on the position, direction, brightness, and / or color associated with each light source.

[0055] Now refer to Figures 3 to 6 Describes a method for combining images to obtain an environment texture.

[0056] Figure 3 is a flow chart showing a method of combining images to obtain environment texture according to an embodiment of the present disclosure. Figures 3 to 6 The description relates to images captured using a front-facing camera and a rear-facing camera of an electronic device, but the images may be captured using any image capture device.

[0057] In the embodiments of the present disclosure, Figure 2 Operation S202 can be performed using Figure 3 In another embodiment of the present invention, an electronic device (eg, electronic device 800) may execute a method for combining images to obtain environment texture.

[0058] Figure 4 The first image captured by using a front camera according to an embodiment of the present disclosure is schematically shown.

[0059] Figure 5 The second image captured using the rear camera according to an embodiment of the present disclosure is schematically shown.

[0060] Figure 6The figure schematically shows an environment texture acquired by combining a first image and a second image according to an embodiment of the present disclosure.

[0061] As described above, the first image and the second image can be captured using a front camera and a rear camera in the electronic device. In some electronic devices, one or both of the front camera and the rear camera are capable of capturing a full 180-degree field of view, for example using a suitable wide-angle lens, such as a fisheye lens. In such an embodiment, the resulting image can be copied directly into an equirectangular environment texture with little or no post-processing.

[0062] However, in many electronic devices, the front camera and the rear camera may only be able to capture rectilinear images, so that one or both of the first image and the second image may be rectilinear images. For example, the integrated cameras in many smart phones are equipped with rectilinear lenses, which means that light is captured onto a flat surface. Examples of the rectilinear first image 401 and the rectilinear second image 501 are shown in FIG. Figure 4 and Figure 5 See Figure 4 , the first image 401 is a rectilinear image of an indoor environment captured by a front camera. The first image 401 includes a floor 411, a wall 412, a ceiling 413, and a first foreground object 414. Figure 5 , the second image 501 is a rectilinear image of the same indoor environment captured by the rear camera. The second image 501 includes a floor 511, a wall 512, a ceiling 513, and a second foreground object 514. Since the rectilinear image includes a field of view of less than 180 degrees, different areas of the floor 411, 511, the walls 412, 512, and the ceiling 413, 513 are visible in each image 401, 501.

[0063] In some embodiments where one or both of the first image and the second image are rectilinear images, post-processing may be applied to the rectilinear images when generating the environment texture. The post-processing operation may compensate for the fact that the captured first image and the second image are not in the correct format for use in the 360-degree environment texture, and may estimate texture information for areas of the environment that are not visible in the first image or the second image.

[0064] Reference Figure 3 , at operation S301, the electronic device may deform the rectilinear first image 401 and / or the rectilinear second image 501 according to a spherical projection. Figure 6 Shown by combining Figure 5 The straight line first image 401 and Figure 6An example of an equirectangular environment texture 601 obtained by respectively deforming a rectilinear second image 501. In some embodiments where one of the first image and the second image includes a 180 degree field of view (e.g., a fisheye image), the deformation may only need to be applied to the other of the first image and the second image, i.e., one of the rectilinear images in the first image and the second image. The deformation applied at operation S301 can be configured to project the rectilinear images 401 and 501 onto the surface of a virtual sphere surrounding the user. For example, a sphere centered on the electronic device used to capture the first image 401 and the second image 501. The deformed images 401 and 501 are then joined side by side to obtain an approximation of the equirectangular environment texture 601, as shown in FIG. Figure 6 shown.

[0065] At operation S302, the electronic device may fuse the edges of the first image and the second image. The adjacent edges of the first image 401 and the second image 501 used to create the environmental texture are fused by mixing the information from the two images. Specifically, the deformed first image 401 and the second image 501 are arranged side by side in an equirectangular environmental texture, and the information from the two images is mixed in the vertical area on either side of the environmental texture 601 and along the vertical center line where the two images 401 and 501 meet. Since each of the first image 401 and the second image 501 covers a field of view of less than 180 degrees, depending on the scene captured within the image, joining the images side by side without fusion may cause discontinuities due to blind spots between the images. Fusion of the adjacent edges of the first image 401 and the second image 501 can help soften the transition between the images 401 and 501 and reduce the prominence of visual artifacts caused by discontinuities.

[0066] At operation S303, the electronic device may generate a texture map (mipmap) from the first image and / or the second image. At operation S304, the electronic device may acquire texture information of an unfilled area of ​​an environment texture based on a texture map image of a lower resolution.

[0067] Reference Figure 6 , the approximately equirectangular environment texture 601 may include a plurality of unfilled areas 611 near the corners of the deformed first image 401 and the second image 501. Figure 6 , a plurality of unfilled regions 611 are shown using hatching. The plurality of unfilled regions 611 appear because the deformed first and second images only partially fill respective corresponding regions of the environment texture. This occurs because the equirectangular environment texture 601 represents a full 360-degree field of view, while the field of view of each rectilinear image 401, 501 is less than 180 degrees. Therefore, Figure 6The multiple unfilled areas 611 in the environment constitute blind spots in the environment, which are currently not visible to the front camera or the rear camera.

[0068] According to an embodiment of the present disclosure, missing texture information of a plurality of unfilled regions 611 of the environment texture 601 is estimated based on a lower resolution version of one of the first image 401 and the second image 501. Figure 6 , texture information of a plurality of unfilled areas 611 at the lower right corner of the equirectangular environment texture 601 can be estimated based on the lower resolution version of the second image 501. In the present embodiment, a texture map including a plurality of lower resolution versions is generated based on each of the first image 401 and the second image 501. In the texture map, each version of the original image may have a height and a width that are half the height and the width of the image in the previous texture map level.

[0069] The lower resolution version may be enlarged to a size larger than the original image, and then the same deformation may be applied to the enlarged low resolution image. By increasing the size of the image, the deformed low resolution version is larger than the deformed original version, so objects within the image occupy a larger area of ​​the environment texture 601, including the plurality of unfilled regions 611. Pixel data from the deformed low resolution version may then be used to fill in missing texture information in the plurality of unfilled regions 611, which, as described above, are areas that were not filled by the original deformed image.

[0070] Filling in the lost texture information using an enlarged low-resolution version of the original image 401, 501 has the effect of blurring features in the plurality of unfilled regions 611. However, in an equirectangular projection, objects near the top and bottom of the field of view appear stretched and elongated relative to objects near the horizontal centerline. Therefore, using an enlarged low-resolution version can simulate the deformation that naturally occurs in an equirectangular projection. In addition, the loss of spatial information in the plurality of unfilled regions 611 may not have a significant impact on the final rendered product because texture information away from the central axis will be automatically compressed during the rendering process.

[0071] According to an embodiment of the present disclosure, obtaining unfilled texture information may include obtaining texture information of multiple unfilled areas at different distances from the center of the deformed first image 401 and / or the second image 501 based on multiple lower resolution versions of the corresponding first image 401 or second image 501. For example, when texture mapping is used, images for different texture mapping levels can be used to estimate missing texture information at different distances from the center of the image. The resolution of the version used to obtain texture information can gradually decrease as the distance from the center increases.

[0072] Although texture mapping is used in the embodiment of the present disclosure, according to the embodiment of the present disclosure, the first image 401 or the second image 501 may be converted to any lower resolution for acquiring lost texture information instead of generating texture mapping.

[0073] According to an embodiment of the present disclosure, the electronic device may operate as an HMD. The electronic device may acquire an image of an area in the environment displayed on the HUD. In order to acquire the environment texture, the electronic device may perform a single image acquired to be displayed on the HUD. Figure 3 According to an embodiment of the present disclosure, operation S302 may be omitted.

[0074] Figure 7 is a flow chart illustrating a method of determining the position of one or more light sources according to an embodiment of the present disclosure. Figure 7 The flowchart shown in Figure 2 An example of a method used in operation S203.

[0075] Reference Figure 7 , at operation S701, the electronic device may acquire a threshold image of the environment. For example, in order to acquire a threshold image of the environment, threshold processing may be applied to the image of the environment. Here, threshold processing refers to a process whereby a pixel in the original image is set to have one of two values ​​based on a threshold brightness value (e.g., to create a binary image). For example, if the brightness value of a pixel in the original image is below a certain threshold, the pixel may be assigned as a black pixel. Conversely, if the brightness value of a pixel in the original image falls above a threshold, the pixel may be assigned as a white pixel. In an embodiment of the present disclosure, if the original image is a color image, the image may be first converted to a grayscale image before applying threshold processing.

[0076] In an embodiment of the present disclosure, the threshold may be a fixed threshold that is applied in all cases. However, a fixed threshold does not take into account the actual lighting level in the current environment and may miss light sources or produce false positive values. In another embodiment of the present disclosure, an adaptive threshold may be used, such as by processing the entire image using Otsu's method to find an appropriate threshold based on the color characteristics of the image.

[0077] According to an embodiment of the present disclosure, a blur filter is applied to the image before thresholding. Blurring the input image removes high frequencies and makes it easier to detect light sources by amplifying each light source in the blurred image. Blurring can also cause multiple closely spaced light sources to be merged into a single cluster of pixels. This in turn can allow the group of closely spaced light sources to be modeled as a single light source, which can still provide an acceptable approximation to the final output image while simplifying the rendering process.

[0078] Fig. 9 An image of an environment including multiple light sources is shown in accordance with an embodiment of the present disclosure. Fig.10 An image of an environment including a plurality of light sources after blurring according to an embodiment of the present disclosure is shown, and Fig.11 An image of an environment including multiple light sources after thresholding is shown in accordance with an embodiment of the present disclosure.

[0079] Reference Fig. 9 , environment 901 may include a plurality of light sources 911, 912, and 913. In an embodiment of the present disclosure, each of the plurality of light sources 911, 912, 913 may be associated with a location or position and / or an intensity or brightness level.

[0080] Reference Fig.10 , can be accomplished by making an image of an environment including multiple light sources (e.g. Fig. 9 The image 901 is blurred to obtain a blurred image 1001. Fig.11 , can be obtained by blurring the image of an environment including multiple light sources (e.g. Fig.10 Image 1001) applies threshold processing to obtain image 1101.

[0081] Reference Figure 7 At operation S702, the electronic device may identify a pixel cluster having a brightness value greater than the applied threshold as a light source. According to an embodiment of the present disclosure, a separate light source may be defined for each cluster, or multiple closely spaced clusters may be merged into a single light source to simplify the rendering process. Spatial coordinates may be assigned to each light source based on the position of the corresponding cluster in the image. For example, the geometric center of the bright pixels in each cluster is used as the position of the corresponding light source 1111, 1112, 1113, as shown in FIG. Fig.11 shown.

[0082] At operation S703, the electronic device may determine the relative intensity of the light source based on the size of the cluster. According to an embodiment of the present disclosure, the relative intensity of two or more light sources is determined based on the number of pixels associated with the corresponding cluster of each light source. The light source corresponding to the cluster with more pixels is determined to have a higher intensity than the light source corresponding to the cluster with fewer pixels, so that the brighter light source will produce a larger bright pixel cluster in the threshold image. According to an embodiment of the present disclosure, each light source can be set to have the same default brightness value. This can provide an acceptable approximation, especially in an environment where the lighting level is fairly constant throughout the scene.

[0083] According to an embodiment of the present disclosure, at least one of the direction or the color of the light source may be identified by using an image acquired separately from an image for identifying the position of the light source.

[0084] According to an embodiment of the present disclosure, Figure 7 In the method, the image used to detect the light source can be Figure 2 For example, the first image 401 and the second image 501 for acquiring the environmental texture 601 may be captured using a balanced exposure value suitable for the current environment, which may be automatically calculated by the camera or manually set by the user.

[0085] According to an embodiment of the present disclosure, a light source can be detected from different images of an environment. Figure 2 The image used for light source detection at operation S203 of the present invention may be an image of the environment captured at a lower exposure setting for the first image 401 and the second image 501. Using a lower exposure value has the effect of darkening the image, making it easier to separate and detect the individual light sources. For example, the third image and the fourth image may be captured at a lower exposure setting using the front camera and the rear camera, and then the same exposure settings as above may be used. Figures 3 to 6 The third image and the fourth image are deformed and combined in a similar manner to the described method to obtain a low-exposure image of the environment in an equirectangular image format.

[0086] According to an embodiment of the present disclosure, an electronic device may operate as an HMD. The electronic device may acquire an image of an area in an environment displayed on a HUD. The electronic device may acquire an environmental texture from the acquired image displayed on the HUD. The electronic device may determine the position of one or more light sources based on the acquired image. For example, the position and / or main direction of the light source is determined based on parameters related to one or more light sources directly identified in the acquired image, such as the brightness of each light source, the size of each light source, etc. In addition, the position of the light source that does not exist in the acquired image among the light sources present in the environment is determined based on the following parameters. For example, by identifying the direction of the shadow of an object in the acquired image, the size and / or brightness of the shadow of an object that does not exist in the acquired image, the position, size and / or brightness of the light source may be identified. In addition, while identifying the above parameters, the color of each light source that is in the acquired image and does not exist in the acquired image is identified together. The electronic device may present an augmented reality virtual object based on the acquired environmental texture and one or more light sources whose position, direction, brightness and / or color are identified. According to an embodiment of the present disclosure, at least one of the direction or color of the light source may be identified by using a single image acquired separately from the image used to identify the position of the light source.

[0087] Figure 8An electronic device for presenting an augmented reality object according to an embodiment of the present disclosure is schematically shown. The device may be configured to perform any of the methods disclosed herein, such as the above reference Figure 2 , Figure 3 and Figure 7 The method described.

[0088] Reference Figure 8 , the electronic device 800 includes a processing unit 801 for executing computer program instructions and a memory 802 arranged to store the computer program instructions. The memory 802 may include any suitable computer-readable storage medium capable of storing a computer program product including computer program instructions executed by the processing unit 801. When executed by the processing unit 801, the computer program instructions may cause the electronic device 800 to perform any method disclosed herein.

[0089] According to an embodiment of the present disclosure, the electronic device 800 further includes a front camera 803, a rear camera 804, a display controller 805, and a display unit 806. For example, the electronic device 800 may be implemented as a smart phone, a tablet computer, or other electronic devices including the front and rear cameras 803, 804 and a suitable display unit 806. The display controller 805 is configured to control the display unit 806 to display a composite image including the presented augmented reality virtual object.

[0090] Despite Figure 8 Although not shown in the figure, the electronic device 800 also includes one or more sensors that can measure physical quantities or detect the activation state of the electronic device 800 and convert the measured or detected information into electrical signals. For example, the sensor may include a depth or distance sensor, a color sensor (e.g., a red-green-blue (RGB) sensor), etc. The sensor may also include a control circuit for controlling at least one of the sensors included therein. Any of these sensors may be located within the electronic device 800.

[0091] According to an embodiment of the present disclosure, the electronic device 800 may be installed in a wearable device. For example, the electronic device 800 may operate as an HMD and run an augmented reality mode.

[0092] According to an embodiment of the present disclosure, the processing unit 801 and the memory 802 may be implemented in a physical device separated from one or more of the front camera 803, the rear camera 804, the display controller 805, and the display unit 806. For example, the processing unit 801 and the memory 802 may be implemented as a server, which is arranged to communicate remotely with an electronic device including the front camera 803, the rear camera 804, the display controller 805, and the display unit 806, so that the processing of acquiring environmental texture, determining light sources, and / or presenting augmented reality virtual objects can be performed remotely. When the electronic device itself has only limited processing capabilities, remote processing may be advantageous. According to an embodiment of the present disclosure, the synthesized image can be output to a physically separated display unit, in which case the electronic device 800 does not need to include the display unit 806.

[0093] According to an embodiment of the present disclosure, an environmental texture is acquired based on images captured using a front camera and a rear camera of an electronic device. Then, the environmental texture is used when presenting an augmented reality virtual object. The method provides a computationally efficient solution that does not require any input or parameterization from a user before execution and is suitable for implementation using electronic device hardware of the related art. Therefore, the above-mentioned processing can be performed substantially in real time, thereby allowing the augmented reality presentation processing to take into account dynamic changes in lighting conditions.

[0094] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for operating an electronic device for presenting an augmented reality object, the method comprising: Acquire at least one first image of an environment in which an augmented reality virtual object is to be displayed, the at least one first image comprising at least one straight line image; Acquire an environment texture based on the acquired at least one first image; identifying in at least one second image at least one cluster comprising pixels having brightness values ​​greater than a threshold; determining a position of at least one light source in the at least one second image based on the position of the pixel within the at least one cluster; presenting the augmented reality virtual object based on the acquired environment texture and the determined position of the at least one light source; as well as displaying the rendered augmented reality virtual object on a display of the electronic device, Wherein, acquiring the environment texture based on the acquired at least one first image comprises: deforming the at least one rectilinear image according to a spherical projection; determining an approximation of the texture of the environment based on the deformed at least one rectilinear image; generating a texture map from the at least one first image; and Texture information of at least one unfilled area in the approximation of the environment texture is obtained based on the texture mapping to obtain the environment texture.

2. The method according to claim 1, Wherein acquiring the at least one first image of the environment comprises acquiring an image to be displayed on a display of the electronic device, the display of the electronic device corresponding to a head mounted display device.

3. The method of claim 1 , wherein determining the position of the at least one light source comprises: setting a first value to a first pixel in the at least one second image based on identifying that the brightness value of the first pixel is greater than the threshold; as well as Based on identifying that a brightness value of a second pixel in the at least one second image is less than the threshold, a second value is set to the second pixel.

4. The method of claim 1 , wherein determining the position of the at least one light source comprises: identifying a geometric center point of pixels in each of the at least one cluster; as well as The geometric center point is set as the position of each light source of the at least one light source.

5. The method according to claim 1, further comprising: determining a relative intensity of two or more of the at least one light source based on a number of pixels in each of the at least one cluster, Therein, one light source corresponding to a cluster having more pixels is determined to have a higher intensity than another light source corresponding to a cluster having fewer pixels.

6. The method according to claim 1, further comprising: The at least one second image is acquired based on a lower exposure setting than the at least one first image.

7. The method of claim 2, wherein determining the location of the at least one light source comprises identifying a location of a light source among the light sources in the environment that is outside of the acquired image based on at least one of: the direction of the shadow of the object in the acquired image, the size of the shadow, or Luminance values ​​of pixels associated with the object other than the at least one light source.

8. An electronic device for presenting an augmented reality object, the electronic device comprising: monitor; Image sensor; at least one processor coupled to the display and the image sensor; as well as A memory coupled to the at least one processor, the memory comprising instructions executable by the at least one processor and configured to cause the at least one processor to perform the following operations: Acquire, using the image sensor, at least one first image of an environment in which an augmented reality virtual object is to be displayed, the at least one first image comprising at least one straight line image; identifying in at least one second image at least one cluster comprising pixels having brightness values ​​greater than a threshold, determining a position of at least one light source in the at least one second image based on the position of the pixel within the at least one cluster, presenting the augmented reality virtual object based on the environment texture of the at least one first image and the determined position of the at least one light source, and displaying the rendered augmented reality virtual object on the display, Wherein, acquiring the environment texture based on the acquired at least one first image comprises: deforming the at least one rectilinear image according to a spherical projection; determining an approximation of the texture of the environment based on the deformed at least one rectilinear image; generating a texture map from the at least one first image; and The texture information of at least one unfilled area in the approximation of the environment texture is acquired based on the texture mapping to obtain the environment texture.

9. The electronic device according to claim 8, in, In order to acquire the at least one first image of the environment, the at least one processor is further configured to acquire an image to be displayed on the display of the electronic device by using the image sensor, the display of the electronic device corresponding to a head mounted display device.

10. The electronic device according to claim 8, wherein: To determine the position of the at least one light source, the at least one processor is further configured to: setting a first value to a first pixel in the at least one second image based on identifying that the brightness value of the first pixel is greater than the threshold, Based on identifying that a brightness value of a second pixel in the at least one second image is less than the threshold, a second value is set to the second pixel.

11. The electronic device according to claim 8, wherein: To determine the position of the at least one light source, the at least one processor is further configured to: identifying a geometric center point of pixels in each of the at least one cluster, and The geometric center point is set as the position of each light source of the at least one light source.

12. The electronic device according to claim 8, in, The at least one processor is further configured to determine a relative intensity of one or two of the at least one light source based on the number of pixels in each of the at least one cluster, and Therein, one light source corresponding to a cluster having more pixels is determined to have a higher intensity than another light source corresponding to a cluster having fewer pixels.

13. The electronic device according to claim 9, wherein: To determine the position of the at least one light source, the at least one processor is further configured to identify the position of a light source among the light sources in the environment that is present outside the acquired image based on at least one of the following: the direction of the shadow of the object in the acquired image, the size of the shadow, or Luminance values ​​of pixels associated with the object other than the at least one light source.

14. A computer program product comprising computer program instructions which, when executed by a processor, perform the method of any one of claims 1 to 7.

15. At least one non-transitory computer-readable storage medium arranged to store a computer program product, the computer program product comprising computer program instructions which, when executed by a processor, perform the method of any one of claims 1 to 7.

Images