Enhanced local contrast
By using enhanced local contrast technology on optical perspective devices, displayable parts of virtual content are generated and mixed with virtual content, the problem of difficult display of virtual shadows and negative light characteristics is solved, and the darkening of the intersection of virtual objects and real objects is achieved. The effect enhances the visual experience of virtual reality.
Patent Information
- Application Number
- CN202010478015.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-04-28
- Filing Date
- 2020-05-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-05-29
AI Technical Summary
Existing optical perspective devices have difficulty displaying virtual shadows and negative light features, such as making real content look darker to provide virtual shadow effects for virtual objects.
By using enhanced local contrast technology on optical perspective devices, images of displayable parts of virtual content are generated and by mixing with images of virtual content, increasing contrast to simulate shadows and other natural visual features, realizing virtual objects and real objects Dark effect of crossing.
It realizes displaying virtual shadows and negative light features on optical perspective devices, enables users to perceive the shadow effect of virtual objects, enhances the visual effect of the boundaries and intersections between virtual objects and real objects, and solves the problem that the existing technology cannot display virtual shadows. question.
Smart Images

Figure CN112017126B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 855,124, filed May 31, 2019, which is incorporated herein by reference in its entirety. Technical Field
[0003] The present disclosure relates generally to providing content on electronic devices, and particularly to systems, methods, and devices for providing content on optical see-through devices. Background Art
[0004] Virtual content may be included as part of the content provided on an optical see-through device. For example, a user may see a real table of his or her physical environment through a see-through portion of a display, and also see a virtual cup displayed on or reflected from the same see-through portion of the display. The virtual cup may be positioned and configured so that it appears to the user to rest on the upper portion of the surface of the real table. The appearance of such virtual objects may be more realistic by using virtual shadows and other added realistic features. However, considering the additional light characteristics of many see-through devices, it is usually impossible to display such features. For many such devices, virtual content is displayed on or reflected from a see-through display. The virtual content may be located in front of the real content, and the light from the virtual content may be added to the light reflected from the real content and seen through the device. In such a system, the virtual cup can be made visible because displaying the virtual cup involves adding brightness to the base light of the table. In contrast, it is impossible to display negative light, for example, it is impossible to display virtual content that makes a portion of the table look darker to provide a virtual shadow of a virtual cup. Summary of the invention
[0005] Some implementations provided herein use enhanced local contrast to provide content on an optical see-through device. In some implementations, the enhanced local contrast can be used to provide an apparent reduction in brightness (e.g., a shadow) or other visual effect. For example, the appearance of a virtual shadow of a virtual cup can be provided on a real table in a physical environment, even though the brightness of the table cannot be reduced. The appearance of the shadow can be provided by selectively enhancing the contrast of a relatively "darker" area (e.g., via an optical illusion / effect) where the user cognitively interprets seeing the shadow should be.
[0006] In some implementations, an electronic device uses a processor to execute a method. The method generates virtual content. For example, this may involve generating virtual content that includes both a virtual cup and a virtual shadow of the cup. In some implementations, a target appearance of the virtual content that takes into account the physical environment is determined, for example, a virtual cup on a real table, the virtual cup having a virtual shadow on the table based on real light sources of the physical environment. The target appearance may be determined via an image or model of the physical environment and photo-realistic 3D rendering technology. Generating the virtual content may involve determining an image of all virtual content (e.g., virtual objects plus virtual shadows without a real table), for example, by subtracting an image of the physical environment from the target appearance.
[0007] The method also determines an image of a displayable portion of the virtual content. The displayable portion of the virtual content is the portion of the virtual content that can be displayed on the perspective device. For example, determining the displayable portion may involve determining the portion of the image of all virtual content that meets certain display criteria (e.g., pixels with brightness values greater than zero or some other threshold). Thus, the image of the displayable portion of the virtual content may include only virtual content that has a positive brightness or actual brightness and is therefore displayable on the optical perspective display.
[0008] The method generates enhanced virtual content by enhancing the local contrast of the image of the displayable portion of the virtual content. In some specific implementations, the enhanced virtual content is generated by mixing the image of the displayable portion with the image of all virtual content. Since the image of all virtual content will generally have a greater contrast, the mixing can have the effect of increasing the contrast of the image of the displayable portion. This increased contrast can be used to achieve visual effects that simulate shadows and other natural visual features. The mixing can selectively use certain contrast levels that are relatively more than other contrast levels of each image to selectively increase the contrast while the increase in overall brightness is limited.
[0009] The method displays enhanced virtual content on the perspective device. In some embodiments, the displayed virtual content includes a virtual object and virtual content that provides enhanced contrast and is located on top of real content. The virtual content located on top of the real content can provide a visual effect that simulates the darkening of a portion of the real object or otherwise changes the appearance of the real object. For example, increasing the contrast of a real table can cause the user to perceive a relatively bright portion of the table and a relatively dark portion of the table, such as a shadow, even if the portion of the table is not actually darker than the real table. In other embodiments, the enhanced contrast is used to improve the appearance of the virtual object itself or to improve the appearance of an intersection, contact, or boundary between a virtual object and real content.
[0010] According to some specific implementations, a non-transitory computer-readable storage medium stores instructions, which are computer-executable to perform or cause the performance of any of the methods described herein. According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.
[0012] Figure 1 An exemplary operating environment according to some implementations is illustrated.
[0013] Figure 2 An exemplary controller according to some implementations is illustrated.
[0014] Figure 3 An exemplary see-through device according to some implementations is illustrated.
[0015] Figure 4 Comparison of the target appearance, the environment without augmentation, and the environment with augmentation is illustrated.
[0016] Figure 5 is a flow chart illustrating an exemplary method for providing content.
[0017] Figure 6 Virtual content created with reference to a physical environment is illustrated.
[0018] Figure 7 Various exemplary contrast levels are illustrated.
[0019] Figure 8 Enhanced local contrast providing a desired visual effect is illustrated.
[0020] Fig. 9 The creation of virtual content from a target appearance and a background (corresponding to positive and negative luminance) is illustrated.
[0021] Fig.10 Illustrated Fig. 9 The displayable portion of the virtual content.
[0022] Fig.11 Providing content using augmented virtual content and background is illustrated.
[0023] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION
[0024] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein. Figures 1 to 3 Exemplary implementations involving an optical see-through device, such as an attached handheld display or a head mounted device (HMD), are shown.
[0025] Figure 1 An exemplary operating environment 100 according to some specific implementations is illustrated. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary specific implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and a device 120, one or both of which may be in a physical environment. The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.
[0026] In some implementations, the controller 110 is configured to manage and coordinate the content experience for the user. In some implementations, the controller 110 includes a suitable combination of software, firmware, or hardware. Figure 2 Controller 110 is described in further detail. In some implementations, controller 110 is a computing device that is located locally or remotely relative to physical environment 105.
[0027] In one example, the controller 110 is a local server located within the physical environment 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the controller 110 is communicatively coupled to the device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).
[0028] In some implementations, the device 120 is configured to present content to a user. In some implementations, the device 120 includes a suitable combination of software, firmware, or hardware. Figure 3 Device 120 is described in more detail. In some implementations, the functionality of controller 110 is provided by or combined with device 120, for example, in the case of an add-on handheld device or HMD used as a standalone unit.
[0029] According to some specific implementations, when the user appears in the physical environment 105, the device 120 presents a CGR environment to the user. A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one physical law. For example, the CGR system can detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. In some cases (e.g., for accessibility reasons), the adjustment of features of virtual objects in the CGR environment can be performed in response to a representation of physical movement (e.g., a voice command).
[0030] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.
[0031] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.
[0032] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.
[0033] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.
[0034] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. As used herein, a device in which at least some light of the physical environment can pass through a transparent or translucent display is referred to as an "optical see-through" device.
[0035] Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on the opaque display. The person uses the system to indirectly view the physical environment via the image or video of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display.
[0036] Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.
[0037] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.
[0038] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.
[0039] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment.
[0040] The device may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one specific implementation, the transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.
[0041] Figure 2Examples of controllers 110 according to some implementations are illustrated. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, or similar type interface), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.
[0042] In some implementations, the one or more communication buses 204 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.
[0043] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and a computer-generated reality (CGR) experience module 240.
[0044] Operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks.
[0045] In some specific implementations, the computer generated reality (CGR) module 240 includes a virtual content unit 242, a 3D rendering unit 244, and an enhancement unit 246 and a CGR rendering unit 248. The virtual content unit 242 is configured to generate virtual content in the form of 2D or 3D models, images or other forms. The 3D rendering unit 244 may also be configured to render virtual content. For example, the 3D rendering unit may create or use a 3D model of a physical environment (e.g., a table, a light source, etc.) together with a 3D model of a virtual object (e.g., a virtual cup) included, and generate virtual shadows or other virtual effects. The enhancement unit 246 is configured to generate enhanced virtual content, including virtual content that enhances local contrast in a CGR environment. The CGR rendering unit 248 uses the enhanced virtual content to present a CGR environment, for example, by providing the enhanced virtual content for display on a perspective display of the device 120 or for reflection on a perspective display of the device 120. In some implementations, the CGR rendering unit 248 presents a series of frames on a display of the device 120 , where each frame includes augmented virtual content.
[0046] Although these modules and units are shown as residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of these modules and units may be located in separate computing devices. Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.
[0047] Figure 3Examples of device 120 according to some implementations are illustrated. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.
[0048] In some implementations, one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.
[0049] In some implementations, one or more displays 312 are configured to present content to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), or similar display types. In some implementations, one or more displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, device 120 includes a single display. As another example, device 120 includes a display for each eye of the user.
[0050] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or their subsets, including an optional operating system 330 and a computer-generated reality (CGR) experience module 340.
[0051] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks.
[0052] In some specific implementations, the computer generated reality (CGR) module 340 includes a virtual content unit 342, a 3D rendering unit 344 and an enhancement unit 346, and a CGR rendering unit 348. The virtual content unit 342 is configured to generate virtual content in the form of a 2D or 3D model, image, or other form. The 3D rendering unit 344 is configured to render virtual content. For example, the 3D rendering unit can create or use a 3D model of a physical environment (e.g., a table, a light source, etc.) together with a 3D model of a virtual object (e.g., a virtual cup) included, and generate virtual shadows or other virtual effects. The enhancement unit 346 is configured to generate enhanced virtual content, including virtual content that enhances local contrast in a CGR environment. The CGR rendering unit 348 uses the enhanced virtual content to present a CGR environment, for example, by providing the enhanced virtual content for display on a perspective display of the device 120 or for reflection onto a perspective display of the device 120. In some implementations, the CGR rendering unit 348 presents a series of frames on a display of the device 120 , where each frame includes augmented virtual content.
[0053] also, Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.
[0054] Figure 4 Target appearance image 405, image 415 of the environment without augmentation, and image 420 of the environment with augmentation are illustrated. Specifically, target appearance image 405 includes virtual object 402—virtual coffee cup—and virtual shadow 410 of virtual object 402. In this example, virtual shadow 410 may be generated via 3D rendering technology that simulates the shadow based on a 3D model of the virtual coffee cup and a real or virtual light source. Since virtual shadow 410 cannot be directly displayed on the intended optical see-through device using existing technology, the appearance of the environment will not include the shadow and will appear as depicted in the image of the environment without augmentation 415.
[0055] In contrast, using one or more of the techniques disclosed herein, an environment with virtual content augmentation may appear as depicted in an image of an environment with virtual content augmentation 420. In this example, virtual content is added in front of the areas on either side of virtual object 402 to enhance the local contrast, thereby creating the appearance of shadows at locations 425. In this example, the appearance of shadows at locations 425 is provided without making those areas darker. Instead, the local contrast around those locations 425 results in the appearance of relative darkness at those locations 425.
[0056] Figure 5 Flowchart of an exemplary method for providing content such as a CGR environment. In some implementations, method 500 is performed by a device (e.g., Figure 1-Figure 3 The method 500 may be performed at a mobile device, an HMD, a desktop computer, a laptop computer, a server device, or by multiple devices communicating with each other. In some implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 500 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).
[0057] At block 502, method 500 generates virtual content. For example, this may involve generating virtual content that includes both a virtual cup and a virtual shadow of the cup. In some implementations, a target appearance of the virtual content that takes into account the physical environment is determined, for example, a virtual cup on a real table with a virtual shadow on the table based on real light sources of the physical environment. The target appearance may be determined via an image or model of the physical environment and photo-realistic 3D rendering techniques. Generating the virtual content may involve determining an image of all virtual content (e.g., virtual objects plus virtual shadows without the real table), for example, by subtracting an image of the physical environment from the target appearance.
[0058] Figure 6 6 is a block diagram illustrating virtual content generated based on real light sources of a physical environment. In this example, the user is observing (e.g., via eye 605) the environment through a combiner 615 (e.g., through a perspective display). The combiner 615 can allow the user to see some of the physical environment around the user while also presenting virtual content on a display 610, which is reflected on the combiner 615 and is therefore also viewed on the combiner 615. In this example, the physical environment includes a real table 625 and a real light source 630. Creating virtual content can involve determining a 2D or 3D model of such a real environment and determining the relative position of virtual content (e.g., virtual content 620). Creating virtual content can also involve determining realistic virtual effects, such as virtual shadows 635, based on the relative position of the virtual content 620 and the real light source 630 in the model.
[0059] Return to Figure 5 After generating the virtual content, at block 504, method 500 determines an image of a displayable portion of the virtual content that is capable of being displayed on a perspective device. For example, determining the displayable portion may involve determining a portion of the image of all virtual content that satisfies certain display criteria (e.g., pixels having a brightness value greater than zero or some other threshold). Thus, the image of the displayable portion of the virtual content may include only virtual content that has a positive brightness or actual brightness and is therefore capable of being displayed on an optical perspective display. The image of the displayable portion of the virtual content may exclude negative brightness that cannot be displayed on a perspective device.
[0060] At box 506, the method 500 generates enhanced virtual content by enhancing the local contrast of the image of the displayable portion of the virtual content. In some specific implementations, the enhanced virtual content is generated by mixing or otherwise combining the image of the displayable portion with the image of all virtual content. Since the images of all virtual content will generally have a greater contrast, mixing can have the effect of increasing the contrast to the image of the displayable portion. This increased contrast can be used to achieve visual effects that simulate shadows and other natural visual features. The mixing can selectively use certain contrast levels that are relatively more than other contrast levels of each image to selectively increase the contrast while the increase in overall brightness is limited. In some specific implementations, a first contrast level portion of the image of all virtual content is mixed according to a first mixing factor, and a second contrast level portion of the image of all virtual content is mixed according to a different second mixing factor.
[0061] Figure 7 Various exemplary contrast levels are illustrated. A given input image 705 is decomposed into layers corresponding to different contrast levels using Gaussian difference, for example, layer one 710a, layer two 710b, layer three 710c, layer four 710d, layer five 710e, layer six 710f, layer seven 710g, layer eight 710h. The combination of these layers together produces the input image. When the image of the displayable portion (image 1) is mixed with the image of all virtual content (image 2), each of these images can be decomposed into multiple layers. These images can then be mixed or otherwise combined using techniques that weight the layers of each image differently. In a specific implementation, for example, images are mixed according to the following table:
[0062] layer The percentage of image 1 used for blending The percentage of image 2 used for blending 1 100% 0% 2 100% 0% 3 100% 0% 4 100% 0% 5 10% 90% 6 10% 90% 7 100% 0% 8 100% 0%
[0063] In some implementations, a pyramid blending technique is used. In some implementations, an image is represented using a pyramid representation that decomposes the image into different layers:
[0064]
[0065] The blend can use such a pyramid representation to perform the blending function:
[0066]
[0067] where A and B are images, l is a layer, and λ is a variable that can be chosen to strategically blend certain layers of each of the images.
[0068] Figure 8An enhanced local contrast that provides a desired visual effect is illustrated. In this example, the target appearance 805 includes darker portions that cannot be achieved using added virtual content. Therefore, the displayable portion of the image of the virtual content will not increase in darkness. As shown in image 800, without increasing darkness, the gray background is fully visible and always appears consistent in color. In contrast, by using enhanced virtual content, as shown in image 810, the appearance of darker portions can be provided. In this example, a mix of the displayable portion of the virtual content and all virtual content is used to produce the displayable virtual content, and the virtual content increases the contrast that gives the appearance of the darker portions.
[0069] Scanline graph 815 illustrates how enhanced contrast can be provided by adding only brightness. In this example, the lowest point in scanline graph 815 corresponds to the brightness of the gray background. Brightness is selectively added to other points to increase contrast around the edges of the darker portions and give the center portion the appearance of being darker than the rest. However, as shown in image 820 overlaying the transition region with added contrast, the inner portion 825 is not actually darker than the outer portion 830.
[0070] In some cases, the mixing or other combination of the image of the displayable portion of the virtual content with the image of all the virtual content produces the result that there are still some areas with negative brightness. In such cases, in some specific implementations, adjustments are made to ensure that the enhanced virtual content can be displayed on the perspective device. In some specific implementations, this involves identifying negative boundaries and adding them to the virtual content to ensure that all virtual content has positive brightness. In some specific implementations, the enhanced virtual content is adjusted to avoid negative brightness by brightening only the negative parts, adjusting the enhanced virtual content to transform some or all parts to brighter so that no part has negative brightness, or otherwise adjusting the enhanced virtual content to ensure that the enhanced virtual content does not include content that cannot be displayed. In some specific implementations, all values of the enhanced content (e.g., over the entire image) are increased by the same amount of brightness, for example, so that the most negative pixel is no longer negative.
[0071] return Figure 5 After generating the augmented virtual content by enhancing the local contrast of the image of the displayable portion of the virtual content, at block 508 , method 500 displays the augmented virtual content on the see-through device.
[0072] Figure 9-11 Creation and display of enhanced virtual content with enhanced local contrast is illustrated. Fig. 94 is a block diagram illustrating the creation of virtual content (corresponding to positive and negative brightness) from a target appearance and background. In this example, background 900 is subtracted from target appearance image 405 to generate an image (eg, difference) of all virtual content 905. Fig.10 1 is a block diagram illustrating an image of a displayable portion of virtual content 1000 (e.g., difference value>0). The image of the displayable portion of virtual content 1000 (e.g., difference value>0) and the image of all virtual content 905 (e.g., difference value) are mixed to generate enhanced virtual content 1100. Fig.11 , the augmented virtual content 1100 is combined with the background 900 to provide the environment with augmentation 420. The environment with augmentation 420 provides a visual appearance similar to the target appearance image 405 without the need to add negative brightness to give the appearance of a shadow.
[0073] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.
[0074] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.
[0075] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages may be used to implement the teachings contained herein in software for programming or configuring a computing device.
[0076] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.
[0077] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values may in practice be based on additional conditions or values beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.
[0078] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0079] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, objects or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components or their groupings.
[0080] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
[0081] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.
Claims
1. A method for displaying content, comprising: At an electronic device having a processor and a see-through display: Generate virtual content, wherein the virtual content includes a first part and a second part, the brightness value of the first part is higher than a threshold brightness value so that the first part can be displayed via a see-through head mounted device HMD, and the brightness value of the second part is lower than the threshold brightness value; determining an image of a displayable portion of the virtual content by selecting the first portion of the virtual content based on the brightness value of the first portion being greater than the threshold brightness value and excluding the second portion of the virtual content based on the brightness value of the second portion being less than the threshold brightness value; generating enhanced virtual content by enhancing a local contrast of the image of the displayable portion of the virtual content; as well as The augmented virtual content is displayed on the see-through display. 2 . The method of claim 1 , wherein generating the virtual content comprises determining a target appearance of the virtual content based on a physical environment. 3 . The method of claim 2 , wherein determining the target appearance of the virtual content comprises determining a virtual shadow of a virtual object based on a light source in the physical environment. 4 . The method of claim 2 , wherein determining the target appearance of the virtual content comprises determining a three-dimensional rendering of the virtual content using photo-realistic 3D rendering techniques and an image or model of the physical environment. 5 . The method of claim 2 , further comprising determining an image of all of the virtual content by subtracting an image of the physical environment from the target appearance. The method of claim 1 , wherein determining the image of the displayable portion of the virtual content comprises excluding portions of the virtual content corresponding to negative brightness. 7 . The method of claim 1 , wherein generating the augmented virtual content comprises blending the image of the displayable portion with images of all of the virtual content. The method of claim 7 , wherein the blending is based on a contrast level.
9. The method according to claim 8, wherein: blending a first contrast level portion of the image of all of the virtual content according to a first blending factor; as well as blending a second contrast level portion of the image of all of the virtual content according to a second blending factor, The first mixing factor is different from the second mixing factor.
10. The method of claim 7, wherein the blending includes adjusting to ensure that augmented virtual content can be displayed on the see-through display.
11. A system for displaying content, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: Generate virtual content, wherein the virtual content includes a first part and a second part, the brightness value of the first part is higher than a threshold brightness value so that the first part can be displayed via a see-through head mounted device HMD, and the brightness value of the second part is lower than the threshold brightness value; determining an image of a displayable portion of the virtual content by selecting the first portion of the virtual content based on the brightness value of the first portion being greater than the threshold brightness value and excluding the second portion of the virtual content based on the brightness value of the second portion being less than the threshold brightness value; generating enhanced virtual content by enhancing a local contrast of the image of the displayable portion of the virtual content; as well as The augmented virtual content is displayed on a see-through display. 12 . The system of claim 11 , wherein generating the virtual content comprises determining a virtual shadow of a virtual object based on a light source in a physical environment.
13. The system of claim 11, wherein the operations further comprise determining an image of all of the virtual content by subtracting an image of a physical environment from a target appearance of the virtual content. 14 . The system of claim 11 , wherein determining the image of the displayable portion of the virtual content comprises excluding portions of the virtual content corresponding to negative brightness.
15. The system of claim 11, wherein generating the augmented virtual content comprises blending the image of the displayable portion with images of all of the virtual content.
16. The system of claim 15, wherein the blending is based on a contrast level.
17. The system of claim 16, wherein: blending a first contrast level portion of the image of all of the virtual content according to a first blending factor; and blending a second contrast level portion of the image of all of the virtual content according to a second blending factor, The first mixing factor is different from the second mixing factor.
18. A non-transitory computer-readable storage medium storing program instructions executable on a computer to perform operations comprising: Generate virtual content, wherein the virtual content includes a first part and a second part, the brightness value of the first part is higher than a threshold brightness value so that the first part can be displayed via a see-through head mounted device HMD, and the brightness value of the second part is lower than the threshold brightness value; determining an image of a displayable portion of the virtual content by selecting the first portion of the virtual content based on the brightness value of the first portion being greater than the threshold brightness value and excluding the second portion of the virtual content based on the brightness value of the second portion being less than the threshold brightness value; generating enhanced virtual content by enhancing a local contrast of the image of the displayable portion of the virtual content; as well as The augmented virtual content is displayed on a see-through display.
19. The non-transitory computer-readable storage medium of claim 18, wherein generating the augmented virtual content comprises blending the image of the displayable portion with images of all of the virtual content.
20. The non-transitory computer readable storage medium of claim 19, wherein the blending is based on a contrast level.
21. A method for displaying content, comprising: At an electronic device having a processor: Generate virtual content for inclusion in computer generated reality (CGR) environments; Determine an image of a displayable portion of the virtual content that can be displayed on a see-through head mounted device HMD, wherein the displayable portion is a portion of the virtual content having a brightness value greater than a threshold brightness value; generating enhanced virtual content by enhancing a local contrast of the image of the displayable portion of the virtual content, wherein the enhanced virtual content comprises blending the image of the displayable portion with images of all of the virtual content, wherein the blending is based on a contrast level, wherein: blending a first contrast level portion of the image of all of the virtual content according to a first blending factor; as well as blending a second contrast level portion of the image of all of the virtual content according to a second blending factor, wherein the first blending factor is different from the second blending factor and The CGR environment is presented by displaying the augmented virtual content on the see-through HMD.
