Method, device and program for displaying a 3D representation of an object based on orientation information of a display

Abstract

The present disclosure relates to methods, devices, and programs that display a 3D representation of an object based on orientation information of a display. Techniques are disclosed to display graphical elements in a manner that simulates three-dimensional (3D) visibility, including parallax and shading. More specifically, a plurality of images can be used to construct a lighting model of a target object, each captured image having a known spatial relationship to the target 3D object. For example, in one embodiment, this can be accomplished using polynomial texture mapping (PTM) with spherical or hemispherical harmonics. Using the PTM technique, a relatively small number of base images can be identified. When the target object is to be displayed, a combination of the base images can be generated using orientation information in order to simulate a 3D representation of the target object.

Description

[0001] This application is a divisional of the invention patent application No. 201680057214.X entered into the Chinese national phase on September 23, 2016, entitled "Method, apparatus and program for displaying a 3D representation of an object based on orientation information of a display", the disclosure of which is incorporated herein by reference in its entirety. BACKGROUND

[0002] Actual display of a three-dimensional (3D) object on a two-dimensional (2D) surface has been a long-term goal in the field of image processing. One approach to simulate a 3D object is to take a large number of images each illuminated from a different position. Then, a particular image can be selected and displayed based on the detected light source position (e.g., through an ambient or color light sensor). Another approach is to take a large number of images each with a 3D object at a different position relative to a fixed light source. Again, a particular image can be selected and displayed based on the determined orientation of the 3D object (e.g., through the use of an accelerometer). Another approach combines the two approaches above, such that both the light position and the object orientation can be taken into account. It should be relatively easy to grasp that the number of images required in either of the first two approaches can become very large - making it difficult to implement in low memory devices. SUMMARY

[0003] In one embodiment, the disclosed concept provides a method of displaying a three-dimensional (3D) representation of an object based on orientation information. The method includes displaying a first image of the object on a display unit of an electronic device, wherein the first image is indicative of a first 3D representation of the object; determining orientation information of the electronic device (based on output from one or more sensors integral with the electronic device); determining a second image to be displayed based on a light model of the object and the orientation information; adding a synthetic shadow to the second image based on the orientation information to generate a third image; and displaying the third image of the object on the display unit, wherein the third image is indicative of a second 3D representation of the object - the second 3D representation being different from the first 3D representation.

[0004] In one embodiment, orientation information can be determined relative to a gravitational field using, for example, an accelerometer or gyroscope. In another embodiment, the orientation information can be based on the direction of light. In another embodiment, an image can be captured at a time that coincides with the display of a first image (in the direction of light emitted from the display unit). The image can then be analyzed to identify certain types of objects, and the orientation of the electronic device can be determined accordingly. By way of example, if the captured image includes a face, the angle of the face within the captured frame can provide some orientation information. Various types of light models can be used. In one embodiment, the light model can be a polynomial texture mapping (PTM) model. Generally, the model can encode or predict the angle of light, and therefore the representation of the object, based on the orientation information. In addition to synthetic shadows, parallax information can be incorporated into the model or added as synthetic shadows. A computer-executable program for implementing the disclosed method can be stored in any medium that is readable and executable by a computer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A two-stage operation according to one embodiment is shown.

[0006] Figure 2A and 2B Two baseline image capture operations are shown, according to one embodiment.

[0007] Figure 3 A light model system according to one embodiment is shown.

[0008] Figure 4 A light model system according to another embodiment is shown.

[0009] Figure 5 A system according to another embodiment is shown.

[0010] Figure 6 A computer system according to one embodiment is shown. DETAILED DESCRIPTION

[0011] The present disclosure relates to systems, methods, and computer-readable media for displaying graphic elements that exhibit three-dimensional (3D) behavior. Generally, a technique for displaying graphic elements in a manner that simulates full 3D visibility (including parallax and shadows) is disclosed. More specifically, a plurality of captured images, each having a known spatial relationship with a target 3D object, can be used to construct an illumination model of the target object. For example, in one embodiment, this operation can be implemented using polynomial texture mapping (PTM) utilizing spherical harmonics or hemispherical harmonics. Using PTM technology, a relatively small number of base images can be identified. When the target object is to be displayed, a combination of base images can be generated using orientation information to simulate a 3D representation of the target object—in some embodiments, using shadows and parallax distortion. The orientation information can be obtained from, for example, an accelerometer or a light sensor.

[0012] In the following description, for purposes of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the drawings in the present disclosure represent structures and devices in block diagram form to avoid obscuring the novel aspects of the disclosed concepts. For the sake of clarity, not all features of the actual specific implementation are described. In addition, the language used in this disclosure has been selected primarily for readability and instructional purposes and may not have been selected to delineate or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such subject matter. Reference in this disclosure to "one embodiment" or "an embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be understood as necessarily all referring to the same embodiment.

[0013] It will be appreciated that in the development of any actual implementation (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary from one implementation to another. It will also be appreciated that such development work may be complex and time-consuming, but nonetheless, it will be a routine undertaking for those of ordinary skill in the design and implementation of graphics processing system implementations having the benefit of this disclosure.

[0014] See also Figure 1, the technology according to the present disclosure can be considered to consist of a model development phase 100 and a model deployment phase 105. The model development phase 100 may include the capture of baseline images (block 110) and the development of models of these images (block 115). In one embodiment, the model 120 may include multiple images of the target object captured at different viewing positions and / or lighting angles. In another embodiment, the model 120 may include the development of a PTM model based on the captured baseline images. In another embodiment, the model 120 may include a combination of the captured images and one or more PTM models. Once generated, the model 120 may be deployed to an electronic device 125. As shown, the electronic device 125 may include, according to one embodiment, a communication interface 130, one or more processors 135, graphics hardware 140, a display element or unit 145, device sensors 150, memory 155, an image capture system 160, and an audio system 165 that may be coupled via a system bus or a backplane 170 containing one or more continuous (as shown) or discontinuous communication links.

[0015] The communication interface 130 can be used to connect the electronic device 125 to one or more networks. Exemplary networks include, but are not limited to, local networks such as USB or Bluetooth networks, cellular networks, organizational local area networks, and wide area networks such as the Internet. The communication interface 130 can use any suitable technology (e.g., wired or wireless technology) and protocol (e.g., Transmission Control Protocol (TCP), Internet Protocol (IP), User Datagram Protocol (UDP), Internet Control Message Protocol (ICMP), Hypertext Transfer Protocol (HTTP), Post Office Protocol (POP), File Transfer Protocol (FTP), and Internet Message Access Protocol (IMAP)). The one or more processors 135 can be system-on-chips such as those present in mobile devices and include one or more dedicated graphics processing units (GPUs). The processors 135 can be based on a reduced instruction set computer (RISC) architecture or a complex instruction set computer (CISC) architecture or any other suitable architecture, and each processor can include one or more processing cores. The graphics hardware 140 can be dedicated computing hardware for processing graphics and / or assisting the one or more processors 135 in performing computing tasks. In one embodiment, graphics hardware 140 may include one or more programmable GPUs, and each such unit may include one or more processing cores. For example, display 145 may use any type of display technology, such as light-emitting diode (LED) technology. Display 145 may provide input and output devices suitable for device 125. By way of example, device sensors 150 may include a 3D depth sensor, a proximity sensor, an ambient light sensor, an accelerometer, and / or a gyroscope. Memory 155 represents both volatile and non-volatile memory. Volatile memory may include one or more different types of media (typically solid-state) used by one or more processors 135 and graphics hardware 140. For example, memory 155 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Memory 155 may also include one or more non-transitory storage media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and tapes, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). Memory 155 may be used to retain media (e.g., audio, image, and video files), preference information, device profile information, computer program instructions organized into one or more modules and written in any desired computer programming language, and any other suitable data. When executed by one or more processors 135 and / or graphics hardware 140, such computer program code may implement one or more of the techniques or features described herein. Image capture system 160 may capture still and video images and includes one or more image sensors and one or more lens assemblies.The output from the image capture system 160 may be processed, at least in part, by one or more video codecs and / or one or more processors 135 and / or graphics hardware 140, and / or a dedicated image processing unit incorporated within the image capture system 160. Thus, the captured image may be stored in the memory 155. For example, the electronic device 125 may have two main surfaces. The first surface, or front surface, may coincide with the display unit 145. The second surface, or rear surface, may be an opposing surface. In some embodiments, the image capture system 160 may include one or more cameras directed outward from the first surface and one or more cameras directed outward from the second surface. The electronic device 125 may be, for example, a mobile phone, a personal media device, a portable camera, or a tablet, laptop, or desktop computer system.

[0016] Baseline image capture according to block 110 may include one or two phases. Figure 2A , stage-1 200 may include a 3D target object 205 illuminated from a light source 210. As the camera 215 moves along a path 225 from position 220A to position 220B to position 220C, a relatively large number of images may be acquired to generate a stage-1 image corpus 230. For example, when moving from position 220A to position 220C, a total of 180 images may be captured (e.g., 1 per movement). ° In another embodiment, the camera 215 may be moved completely around the target object 205. In this embodiment, a total of 360 images may be captured (e.g., 1 image per movement). ° Image). Reference Figure 2B , optional stage-2 235 includes a 3D target object 205 illuminated by a light source 210 that can move along a path 245 from position 240A to position 240B to position 240C, while a camera 215 remains in a single position to capture a relatively large number of images (e.g., 150) to generate a stage-2 image corpus 250. The exact number of images required to generate the image corpus 230 and the image corpus 250 may depend on the desired fidelity of the resulting model—the more accurate the model, the more images will generally be required. In some embodiments, the captured images 230 and 250 capture shadow, highlight, and parallax information.

[0017] refer to Figure 3In one embodiment, the image corpus 230 may be organized such that each (or at least some) images are associated with their corresponding viewing angle, as shown in table 300. According to this type of embodiment, the mapping between viewing (capture) angles and target objects 205 may be considered a model (e.g., model 120). During runtime (e.g., using electronic device 125), the viewing angle may be determined using sensor device 150. Once determined, the corresponding image may be retrieved from memory 155 and displayed using display element 145. In one embodiment, if the viewing angle determined based on the sensor output is between the viewing angle determined based on the sensor output and the viewing angle determined based on the sensor output, the corresponding image may be retrieved from memory 155 and displayed using display element 145. Figure 2A If the image corpus 230 is captured between two viewing angles indicated by the sensor, then the two images on either "side" of the viewing angle provided by the sensor may be combined, for example, as a weighted sum. (The phrase "on either side" as used herein refers to the image captured associated with the lower viewing angle that is most closely identical to the viewing angle indicated by the sensor and the image captured associated with the higher viewing angle that is most closely identical to the viewing angle indicated by the sensor.) One of ordinary skill in the art will recognize that the image corpus 230 may be maintained in a structure other than a single table, such as Figure 3 For example, multiple tables such as in a relational database or a B-tree or other data structure may be used for data comparison and retrieval operations.

[0018] refer to Figure 4 Based on the model generated in block 115, PTM operations 400 can be applied independently to each image in the image corpus 250 to generate a PTM model 405. During operation (e.g., using electronic device 125), device sensor 150 can be used to determine the illumination angle relative to target object 205 (e.g., electronic device 125). Once determined, the corresponding position can be input into PTM model 405 (e.g., represented by x-position 415 and y-position 420, and optionally z-position (not shown)) and used to generate output image 410. In another embodiment, the illumination angle can be represented in another coordinate system, such as yaw-pitch-roll, for example. In one embodiment, PTM operations 400 can employ spherical harmonics (SH). In another embodiment, PTM operations 400 can employ hemispherical harmonics (HSH). In other embodiments, different basis functions can be used, such as Zernike polynomials, spherical wavelet functions, and Makhotkin hemispherical harmonics, for example. The exact functional relationship or polynomial chosen may be a function of the implemented computing environment, the desired fidelity of the resulting light model, and the amount of memory required for the model.

[0019] One feature of the PTM operation 400 is that the resulting model 405 can use significantly fewer images than those in the image corpus 250. The image corpus 250 may include a relatively large number of high-resolution color images (e.g., each corpus may include 50-400 images). In contrast, the PTM model 405 may require only a small number of "images" to generate all the images within the scope of the model. For example, in one embodiment, the PTM model 405 may employ spherical harmonics and yield a polynomial of the following form.

[0020] p i =a0x 2 +a1y 2 +a2xy+a3x+a4y+a5, Equation 1

[0021] Among them, "p i ” represents the model output for pixel “i” at a given illumination position (x, y), and a0 through a5 are model coefficients whose values ​​are returned or found by the PTM operation 400. In general, the model coefficients a0 through a5 may be different for each pixel of the image 410 represented by the x input 415 and the y input 420.

[0022] In the implementation, p defined by Equation 1 i represents only the intensity or brightness of the ith pixel in the output image 410. To refer to a color, we can refer to the color matrix [C] such that:

[0023] [P] = [C] [P], Equation 2

[0024] where [C] represents the color value associated with each pixel in the output image [P] (e.g., output image 410). In one embodiment, each pixel value in [C] may be the average color value of all corresponding pixels in image corpus 250. In another embodiment, each pixel value in [C] may be the median value of all corresponding pixels in image corpus 250. In another embodiment, each pixel value in the chrominance image [C] may be the weighted average of all corresponding color values ​​in image corpus 250. In another embodiment, the chrominance values ​​from image corpus 250 may be combined in any manner deemed useful for a particular embodiment (e.g., non-linear).

[0025] according to Figure 1 The model deployment phase 105 may be invoked after at least one generated model (e.g., model 300 and / or model 405) is transferred to the memory 155 of the electronic device 125. Once installed on the device 125, the target object 205 may be displayed on the display unit 145. Figure 5, system 500 according to another embodiment may employ device sensors 150 to supply inputs (e.g., 415 and 420) to models 300 and 405. In one embodiment, device sensors 150 may include environmental and / or color sensors to identify the location and temperature of a light source. In another embodiment, device sensors 150 include a gyroscope and / or accelerometer so that the orientation of device 125 can be determined. If models 300 and 405 are used simultaneously, their respective output images may be combined 505 to generate output image 510. In one embodiment, the combining operation 505 may be a simple merging operation. In another embodiment, the combining operation 505 may represent a weighted combination of the outputs of each model. In another embodiment, the combining operation 505 may actually select a model output based on sensor input and / or user input.

[0026] As another example, consider a first scenario where model 405 is operational and device sensor 150 indicates that device 125 is tilted in a direction represented by an observer looking down at target object 205 at approximately a 45° angle. If a person holds an object in their hand and looks directly down at the top of the object, they would expect to see the top surface of the object. As they move their head to a 45° angle, they would expect to see less of the top surface of the object and more of one or more side surfaces. In practice, the 45° angle (expressed in x and y coordinates, see Figure 5 ) is input to the PTM model 405, and the output image 510 will be a combination of PTM coefficient images modified to provide color.

[0027] Image output according to the present disclosure may include shadows, highlights, and parallax to extend this information captured in the generated image corpus. In another embodiment, if shadow information is not included in the image data used to generate the model, synthetic shadows may be generated (e.g., based on image processing) using the tilt and / or orientation of the identified light source (relative to the device 125). Embodiments employing this technique may use sensor input to generate a first output image from a relevant light model (e.g., output image 410 or 510). This image may then be used to generate synthetic shadows. The synthetic shadows may then be applied to the first output image to generate a final output image that may be displayed, for example, on the display unit 145. In another embodiment, the electronic device 125 may include a camera unit facing outward from the display 145. The camera may then capture and analyze images (separately or in combination with the device sensor 150) to determine the device orientation and / or input to the models 300 and 405. The resulting output image (e.g., image 510) may include shadows captured during model generation or synthesized through image analysis. In one embodiment, the captured image may include a face such that various aspects of the detected face (eg, the position of the eyes and / or mouth and / or nose) may be used to determine input for 300 and / or 405 .

[0028] refer to Figure 6 In addition to being deployed on the electronic device 125, the disclosed technology may be developed and deployed on a representative computer system 600 (e.g., a general-purpose computer system such as a desktop, laptop, notebook, or tablet computer system). The computer system 600 may include one or more processors 605, memory 610 (610A and 610B), one or more storage devices 615, graphics hardware 620, device sensors 625 (e.g., a 3D depth sensor, a proximity sensor, an ambient light sensor, a color light sensor, an accelerometer, and / or a gyroscope), a communication interface 630, a user interface adapter 635, and a display adapter 640—all of which may be coupled via a system bus or backplane 645. The processor 605, memory 610 (including storage device 615), graphics hardware 620, device sensors 625, communication interface 630, and system bus or backplane 645 provide communication with the system bus or backplane. Figure 16 or similar functions, and therefore will not be described further. User interface adapter 635 can be used to connect a keyboard 650, a microphone 655, a pointer device 660, a speaker 665, and other user interface devices such as a touchpad and / or a touch screen (not shown). Display adapter 640 can be used to connect one or more display units 670 (similar in function to display unit 145) that can provide touch input capabilities. System 600 can be used to develop models consistent with the present disclosure (e.g., models 120, 300, and 405). Thereafter, the developed model can be deployed to computer system 600 or electronic device 125. (In another embodiment, electronic device 125 can provide sufficient computing power to enable model development, thereby eliminating the need for a general-purpose computer system 600.)

[0029] It should be understood that the above description is intended to be illustrative and not restrictive. The material has been presented to enable any person skilled in the art to make and use the disclosed matter protected by the claims, and the material is provided in the context of specific embodiments, variations of which will be apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in conjunction with each other). For example, the deployment of models 120, 300, and 405 may be developed separately or together. In another embodiment, the image corpora 230 and 250 may be combined and used to generate a single light model. In one or more embodiments, one or more steps disclosed may be omitted, repeated, and / or performed in a different order than described herein. Therefore, the scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the respective terms "comprising" and "wherein."

Claims

1. An electronic device comprising: Memory; a display unit coupled to the memory; orientation sensor element; and One or more processors coupled to the memory, the display unit, and the orientation sensor element, the one or more processors configured to execute program instructions stored in the memory that cause the electronic device to: obtaining orientation information of the electronic device from the orientation sensor element, obtaining an image of the object based on a light model of the object and the orientation information, wherein the light model of the object includes a plurality of images of the object captured from different observation angles, and wherein the obtained image is generated based on one or more of the plurality of images of the object and indicates a three-dimensional representation of the object at a observation angle corresponding to the orientation information of the electronic device, and The obtained image of the object is displayed on the display unit. 2 . The electronic device of claim 1 , wherein the orientation information comprises an orientation of the electronic device relative to a gravitational field. 3 . The electronic device of claim 1 , wherein the program instructions for obtaining the image of the object include program instructions for selecting the image from the plurality of images of the object at different viewing angles. 4 . The electronic device of claim 1 , wherein the program instructions for obtaining the image of the object include program instructions for generating the image based on two or more of the plurality of images of the object at different observation angles.

5. An electronic device according to claim 4, wherein the two or more images among the multiple images include a first image and a second image, wherein the first image and the second image include images in the optical model of the image at an observation angle that most closely corresponds to the orientation information of the electronic device.

6. The electronic device of claim 1 , further comprising program instructions stored in the one or more memory devices that cause the electronic device to: adding a synthetic shading to the obtained image based on the orientation information to generate a modified image of the object; and The modified image of the object is displayed by the display unit. The electronic device of claim 1 , wherein the light model of the object comprises a polynomial texture mapping model. The electronic device of claim 1 , wherein the light model of the object includes parallax information.

9. A computer-readable storage medium comprising instructions stored thereon that, when executed, cause one or more processors to: obtaining orientation information of an electronic device from an orientation sensor element, wherein the electronic device includes a display unit; obtaining an image of the object for display based on a light model of the object and the orientation information, wherein the light model of the object includes a plurality of images of the object captured from different viewing angles, and wherein the obtained image is generated based on one or more of the plurality of images of the object and indicates a three-dimensional representation of the object at a viewing angle corresponding to the orientation information of the electronic device; and The obtained image of the object is displayed on the display unit.

10. The computer-readable storage medium of claim 9, wherein the instructions to cause the one or more processors to obtain orientation information further comprise instructions to cause the one or more processors to determine the orientation information based on a gravitational field.

11. The computer-readable storage medium of claim 9, wherein the instructions for causing the one or more processors to obtain the image of the object further comprise instructions for selecting an image from the plurality of images of the object at different viewing angles.

12. The computer-readable storage medium of claim 9, wherein the instructions for causing the one or more processors to obtain the image of the object further comprise instructions for generating the image based on two or more of the plurality of images of the object at different viewing angles.

13. A computer-readable storage medium according to claim 12, wherein the two or more images of the plurality of images include a first image and a second image, wherein the first image and the second image include images in the optical model of the image at an observation angle that most closely corresponds to the orientation information of the electronic device.

14. The computer-readable storage medium of claim 12, wherein the light model of the object comprises a polynomial texture mapping model.

15. The computer-readable storage medium of claim 9, wherein the light model of the object includes disparity information.

16. A method for displaying a three-dimensional representation of an object, comprising: obtaining orientation information of the electronic device from the orientation sensor element; obtaining an image of the object based on a light model of the object and the orientation information, wherein the light model of the object includes a plurality of images of the object captured from different viewing angles, and wherein the obtained image is generated based on one or more of the plurality of images of the object and indicates a three-dimensional representation of the object at a viewing angle corresponding to the orientation information of the electronic device; as well as The obtained image of the object is displayed on a display unit associated with the electronic device. The method according to claim 16 , wherein the orientation information comprises an orientation of the electronic device relative to a gravitational field or an orientation of the electronic device relative to a light source. The method of claim 16 , wherein the light model of the object comprises a polynomial texture mapping model or disparity information.

19. The method of claim 16, wherein obtaining the image of the object comprises selecting an image from the plurality of images of the object at different viewing angles.

20. The method of claim 16, wherein obtaining the image of the object comprises generating the image based on two or more of the plurality of images of the object at different viewing angles.

21. A method according to claim 20, wherein the two or more images among the multiple images include a first image and a second image, wherein the first image and the second image include images in the optical model of the image at an observation angle that most closely corresponds to the orientation information of the electronic device.

22. The method of claim 16, further comprising: adding a synthetic shadow to the obtained image based on the orientation information to generate a modified image of the object; as well as The modified image of the object is displayed by the display unit.

Images