Video viewing (VST) augmented reality (AR) device and method of operating the vst ar device

Abstract

A method includes receiving a first image and a second image from a first perspective camera having a first camera viewpoint and a second perspective camera having a second camera viewpoint. The method further includes generating a first virtual image corresponding to the first virtual viewpoint by applying the first mapping to the first image. The first mapping is based on a relative position of the first camera viewpoint and a first virtual viewpoint corresponding to a first eye of the user. The method further includes generating a second virtual image corresponding to the second virtual viewpoint by applying the second mapping to the second image. The second mapping is based on a relative position of a second camera viewpoint and a second virtual viewpoint corresponding to a second eye of the user. Further, the method includes displaying the first virtual image to the first virtual viewpoint and the second virtual image to the second virtual viewpoint on at least one display panel of the augmented reality device.

Description

Technical Field

[0001] The present disclosure generally relates to augmented reality (AR) systems and processes. More specifically, the present disclosure relates to methods and devices for direct passthrough in video see-through (VST) AR. Background Art

[0002] Augmented reality (AR) systems can seamlessly blend virtual objects generated by computer graphics into real-world scenes. Optical see-through (OST) AR systems are AR systems in which the user directly views the real-world scene through a head-mounted device (HMD). Unfortunately, OST AR systems face many challenges that may limit their adoption. Some of these challenges include limited field of view, limited use spaces (e.g., indoor use only), inability to display completely opaque black objects, and the use of complex optical pipelines that may require projectors, waveguides, and other optical elements. Summary of the invention

[0003] Technical solutions to technical problems

[0004] The present disclosure provides a method and device for direct pass-through in video see-through (VST) augmented reality (AR).

[0005] According to one aspect of the present disclosure, a method includes: obtaining a first image from a first perspective camera having a first camera viewpoint, and obtaining a second image from a second perspective camera having a second camera viewpoint. The method also includes: generating a first virtual image corresponding to a first virtual viewpoint by applying a first mapping to the first image, wherein the first mapping is based on the relative position of the first camera viewpoint and the first virtual viewpoint corresponding to a first eye of a user. The method also includes: generating a second virtual image corresponding to a second virtual viewpoint by applying a second mapping to the second image, wherein the second mapping is based on the relative position of the second camera viewpoint and the second virtual viewpoint corresponding to a second eye of the user. In addition, the method includes: displaying the first virtual image to the first virtual viewpoint on at least one display panel of an AR device, and displaying the second virtual image to the second virtual viewpoint.

[0006] According to one aspect of the present disclosure, a VST AR device includes at least one display panel, a first perspective camera, a second perspective camera, and at least one processing device. At least one processing device is configured to obtain a first image from a first perspective camera having a first camera viewpoint, and obtain a second image from a second perspective camera having a second camera viewpoint. At least one processing device is also configured to generate a first virtual image corresponding to a first virtual viewpoint by applying a first mapping to the first image, wherein the first mapping is based on the relative position of the first camera viewpoint and the first virtual viewpoint corresponding to the first eye of the user. At least one processing device is also configured to generate a second virtual image corresponding to a second virtual viewpoint by applying a second mapping to the second image, wherein the second mapping is based on the relative position of the second camera viewpoint and the second virtual viewpoint corresponding to the second eye of the user. In addition, at least one processing device is configured to display the first virtual image to the first virtual viewpoint on at least one display panel, and display the second virtual image to the second virtual viewpoint.

[0007] According to one aspect of the present disclosure, a computer-readable medium contains instructions that, when executed, cause at least one processor to perform the method.

[0008] Those skilled in the art can easily see other technical features from the following drawings, descriptions and claims.

[0009] Before proceeding to the following detailed description, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, include both direct and indirect communications. The terms "include" and "comprises," and their derivatives, are inclusive and not limiting. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means include, be included within, be interconnected with, include, be contained within, be connected to or connected with, be coupled to or coupled with, be communicable with, collaborate with, be intertwined, be juxtaposed, be close to, be bound to or bound with, have, have the attributes of, have a relationship to or with, etc.

[0010] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and implemented in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or parts thereof suitable for implementation in appropriate computer-readable program codes. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical disks or erasable memory devices.

[0011] As used herein, terms and phrases such as "having", "may have", "include", or "may include" a feature (such as a number, function, operation, or component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B", and "at least one of A or B" may indicate all of the following situations: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit these components. These terms are only used to distinguish components from each other. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the importance or order of the devices. A first component may be represented as a second component, and vice versa, without departing from the scope of the present disclosure.

[0012] It will be understood that when an element (e.g., a first element) is referred to as being (operably or communicatively) “coupled to” or “connected to” another element (e.g., a second element) or “coupled to” or “connected to” another element (e.g., a second element), it may be coupled to or connected to the other element directly or through a third element. Conversely, it will be understood that when an element (e.g., a first element) is referred to as being “directly coupled to” or “directly connected to” another element (e.g., a second element) or “directly coupled to” or “directly connected to” another element (e.g., a second element), there are no other elements (e.g., a third element) interposed between the element and the other element.

[0013] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "manufactured to," or "capable of...", depending on the scenario. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to." Instead, the phrase "configured to" may mean that a device can perform an operation together with another device or component. For example, the phrase "the processor is configured (or set) to perform A, B, and C" may represent a general-purpose processor (e.g., a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (e.g., an embedded processor) for performing operations.

[0014] As used herein, terms and phrases are used only to describe some embodiments of the present disclosure, without limiting the scope of other embodiments of the present disclosure. It will be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly states otherwise. All terms and phrases (including technical and scientific terms and phrases) used herein have the same meaning as those generally understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It should also be understood that terms and phrases such as those defined in commonly used dictionaries should be interpreted as meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted as ideal or overly formal meanings, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0015] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (e.g., smart glasses, HMD, electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of: a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (e.g., SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (e.g., XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camera, or an electronic photo frame. Still other examples of electronic devices include at least one of the following: various medical devices (e.g., multifunctional portable medical measuring devices (e.g., blood glucose measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resonance angiography (MRA) devices, magnetic resonance imaging (MRI) devices, computed tomography (CT) devices, device imaging, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDRs), flight data recorders (FDRs), automotive infotainment devices, marine electronic devices (e.g., marine navigation devices or gyrocompasses), avionics equipment, security equipment, vehicle head units, industrial or home robots, automated teller machines (ATMs), point-of-sale (POS) devices, or Internet of Things (IoT) devices (e.g., light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters, or kettles). Other examples of electronic devices include at least part of the following: a part of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, and various measuring instruments (e.g., a device for measuring water, electricity, gas, or electromagnetic waves). It should be noted that according to various embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic devices disclosed herein are not limited to the above-mentioned devices, and may include any other electronic devices now known or later developed.

[0016] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (eg, an artificial intelligence electronic device).

[0017] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0018] The description in this application should not be read as implying that any particular element, step, or function is essential that must be included in the claims scope. The scope of the patented subject matter is limited only by the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0020] Figure 1 An example network configuration including an electronic device according to the present disclosure is shown;

[0021] Figure 2A An example process of direct pass-through conversion in video see-through (VST) augmented reality (AR) according to the present disclosure is shown;

[0022] Figure 2B An example process of direct pass-through conversion in video see-through (VST) augmented reality (AR) according to the present disclosure is shown;

[0023] Figure 3 shows an example arrangement between first and second perspective cameras and first and second virtual cameras according to the present disclosure;

[0024] Figure 4A shows an example geometry of direct pass-through with constant depth in VST AR according to the present disclosure;

[0025] Figure 4B shows an example geometry of direct pass-through with constant depth in VST AR according to the present disclosure;

[0026] Figure 5 An example stereoscopic view geometry for direct pass-through conversion according to the present disclosure is shown;

[0027] Figure 6 illustrates an example overlap compensation between a first virtual image and a second virtual image according to the present disclosure;

[0028] Figure 7illustrates example coordinate relationships in overlapping regions between virtual camera frames according to the present disclosure;

[0029] Fig. 8A shows example compensation of dis-occluded areas in a stereo image pair according to the present disclosure;

[0030] Figure 8B shows example compensation of unoccluded areas in a stereo image pair according to the present disclosure;

[0031] Fig. 9A shows example un-occlusion based on moving the camera away from an object in accordance with the present disclosure;

[0032] Fig. 9B An example deocclusion based on moving the camera away from an object according to the present disclosure is shown;

[0033] Fig. 9C An example deocclusion based on moving the camera away from an object according to the present disclosure is shown;

[0034] Fig.10 An example method of direct pass-through in a VST AR according to the present disclosure is shown. DETAILED DESCRIPTION

[0035] The following will describe with reference to the accompanying drawings Figures 1 to 10 However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or substitutes also belong to the scope of the present disclosure.

[0036] In various implementations, the perspective camera is typically a high-resolution camera (e.g., a 2K or 4K camera or higher resolution). In order to provide a quality user experience through an AR headset, the latency of video frame conversion may need to be as low as possible to reduce or prevent the user from noticing the delay when moving the head. However, existing techniques are generally unable to process high-resolution images from a perspective camera to generate virtual view frames with sufficiently low latency.

[0037] Unlike optical see-through AR in which the user can directly see the surrounding scene, video see-through AR uses a see-through camera mounted on an AR headset to reproduce the surrounding scene. Since the position of the see-through camera is different from the position of the user's eyes, the virtual image at the viewpoint of the user's eyes is generated by image frames captured at the viewpoint of the see-through camera. In this way, the user's eyes can see the scene outside through the camera, just as if the see-through camera was mounted at the viewpoint of the eye. The present invention provides a solution for converting high-resolution images from a see-through camera to a virtual camera with low latency, which can be referred to as direct see-through video frame conversion. Direct see-through video frame conversion provides more accurate visualization of the surrounding scene, which enhances the user experience, especially when interacting with objects in the surrounding scene.

[0038] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown in FIG. 1 is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0039] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display panel 160, a communication interface 170, and a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components or may add at least one other component. The bus 110 includes a circuit for connecting components 120 to 180 to each other and transmitting communications (e.g., control messages and / or data) between the components.

[0040] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In some embodiments, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 is capable of controlling at least one other component of the electronic device 101 and / or is capable of performing operations or data processing related to communication or other functions. As described below, the processor 120 can obtain and process input images to perform direct pass-through to support at least one VST AR application, as described in more detail below.

[0041] The memory 130 may include a volatile memory and / or a non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or a program 140. The program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or application) 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0042] The kernel 141 can control or manage system resources (e.g., bus 110, processor 120, or memory 130) used to perform operations or functions implemented in other programs (e.g., middleware 143, API 145, or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145, or application 147 to access various components of the electronic device 101 to control or manage system resources. Application 147 may include one or more applications that, among other things, obtain and process input images to perform direct pass-through to support at least one VST AR application. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions.

[0043] For example, the middleware 143 may act as a relay to allow the API 145 or the application 147 to communicate data with the kernel 141. A plurality of applications 147 may be provided. The middleware 143 is capable of controlling work requests received from the applications 147, for example, by assigning a priority for using system resources (e.g., the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the plurality of applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (e.g., command) for archive control, window control, image processing, or text control.

[0044] The input / output (I / O) interface 150 serves as an interface that can transmit commands or data input from a user or other external devices to other components of the electronic device 101, for example. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0045] The display panel 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display panel 160 may also be a depth perception display, such as a multi-focus display. The display panel 160 is capable of displaying, for example, various contents (e.g., text, images, videos, icons, or symbols) to the user. The display panel 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hovering input by using an electronic pen or a body part of the user.

[0046] The communication interface 170 can, for example, establish communication between the electronic device 101 and an external electronic device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals.

[0047] Wireless communication can use, for example, at least one of Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM) as a cellular communication protocol. Wired connections can include, for example, at least one of universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). Network 162 or 164 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network.

[0048] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101, and convert the measured or detected information into electrical signals. For example, the sensor 180 includes one or more cameras or other imaging sensors (e.g., perspective cameras), which can be used to capture images of scenes. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (e.g., red, green, and blue (RGB) sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. In addition, the sensor 180 may include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor 180 may include a control circuit for controlling at least one sensor included here. Any of these sensors 180 may be located within the electronic device 101 .

[0049] In some embodiments, the electronic device 101 may be a wearable device or a wearable device (e.g., HMD) on which the electronic device can be mounted. For example, the electronic device 101 may represent an XR wearable device such as a head-mounted receiver or smart glasses. In other embodiments, the first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (e.g., HMD) on which the electronic device can be mounted. In other embodiments, when the electronic device 101 is mounted in the first external electronic device 102 (e.g., HMD), the electronic device 101 may communicate with the first external electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the first external electronic device 102, thereby communicating with the first external electronic device 102 without going through a separate network.

[0050] The first external electronic device 102, the second external electronic device 104, and the server 106 may each be a device of the same or different type as the electronic device 101. According to some embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to some embodiments of the present disclosure, all or part of the operations performed on the electronic device 101 may be performed on another one or more other electronic devices (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106). In addition, according to some embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform a certain function or service, the electronic device 101 may request another device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106) to perform at least some functions related thereto, rather than performing the function or service alone or in addition. Another electronic device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106) is capable of performing the requested function or additional function, and transmitting the execution result to the electronic device 101. The electronic device 101 may provide the requested function or service as is or by performing additional processing on the received result. To this end, for example, cloud computing, distributed computing, or client-server computing techniques may be used. Figure 1 The display electronic device 101 includes a communication interface 170 to communicate with the second external electronic device 104 or the server 106 through the network 162 or 164 , but according to some embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0051] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one operation (or function) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may obtain and process input images to perform direct pass-through to support at least one VST AR application, as described in more detail below.

[0052] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various modifications may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0053] Figure 2A An example process of direct pass-through conversion in video see-through (VST) augmented reality (AR) according to the present disclosure is shown. Figure 2B An example process of direct pass-through conversion in video see-through (VST) augmented reality (AR) according to the present disclosure is shown.

[0054] For ease of explanation, Figure 2A and Figure 2B The process 200 is described as using Figure 1 The electronic device 101 is executed by the process 200. However, the process 200 may be used with any other suitable device and in any other suitable system.

[0055] like Figure 1 and Figure 2A As shown, the electronic device 101 includes a first high-resolution perspective camera 202 and a second high-resolution perspective camera 204. For example, the electronic device 101 may include one or more sensors 180 for implementing the high-resolution perspective cameras 202 and 204. In some embodiments, the first high-resolution perspective camera 202 and the second high-resolution perspective camera 204 may be arranged as a left high-resolution perspective camera and a right high-resolution perspective camera. The perspective cameras 202 and 204 may capture images with any suitable resolution and size according to the capabilities of the perspective cameras 202 and 204. The image may also include image data in any suitable format. In some embodiments, for example, the image includes RGB image data, which typically includes image data in three color channels (i.e., red, green, and blue channels). However, the image may include image data in any other suitable form or arrangement.

[0056] The layout of perspective cameras 202 and 204 can be fixed on electronic device 101 and thus have a specified layout and configuration 206. For example, the position and orientation of perspective cameras 202 and 204 can be fixed relative to specific components of electronic device 101 or a specified position of electronic device 101. In some embodiments, the spacing of perspective cameras 202 and 204 can be based on the average human pupil distance (IPD) or the distance between the left eye and the right eye of a user. In other embodiments, the spacing can be customized, for example, based on the specific pupil distance of a given user.

[0057] The electronic device 101 may perform an obtain camera position operation 208 to obtain the positions of the perspective cameras 202 and 204. The obtain camera position operation 208 may also obtain the position of the virtual camera. For example, the electronic device 101 may detect the position of the user's eyes to determine the position of the virtual camera. The electronic device 101 may also perform a build camera mapping operation 210 to determine the relationship between the first perspective camera 202 and the second perspective camera 204 and the first virtual camera and the second virtual camera. For example, the first mapping and the second mapping may be generated based on the parameters and configurations of the first perspective camera 202 and the second perspective camera 204 and the first virtual camera and the second virtual camera. As a specific example, the first mapping may be generated to determine the geometric configuration between the first high-resolution perspective camera 202 and the first virtual camera, and the second mapping may be generated to determine the geometric configuration between the second high-resolution perspective camera 204 and the second virtual camera.

[0058] Due to the separation between the first perspective camera 202 and the second perspective camera 204, the perspective cameras 202 and 204 can capture the scene from different perspectives, which enables each of the first image and the second image to capture details that may be occluded by the other image. For example, the second high-resolution perspective camera 204 can capture features that are occluded by an object from the perspective of the first perspective camera 202, and the first perspective camera 202 can capture features that are occluded by the same object from the perspective of the second perspective camera 204. The electronic device 101 can perform a construct overlap area map operation 212 to identify overlapping areas of the images captured from the first perspective camera 202 and the second perspective camera 204 and the presentation of the first virtual camera and the second virtual camera.

[0059] The perspective cameras 202 and 204 can be used to perform a first image capture operation 214 and a second image capture operation 216, respectively, to obtain a first image and a second image. In some embodiments, the first image and the second image can be a left image and a right image captured from a left high-resolution perspective camera and a right high-resolution perspective camera. The electronic device 101 can also perform de-distortion and correction operations 218 and 220 on the first image and the second image obtained from the image capture operations 214 and 216. For example, the image may be distorted due to the physical or operational characteristics of the perspective cameras 202 and 204 that capture the image, and each de-distortion and correction operation 218 and 220 can operate to distort or otherwise reduce or eliminate these distortions in the image. In some embodiments, the de-distortion and correction operations 218 and 220 can access the distortion model and the imaging sensor calibration parameters, and can use this information to perform image de-distortion. The distortion model represents a model that knows how the perspective cameras 202 and 204 distort the image, which (in some cases) can be generated during the calibration process of the perspective cameras 202 and 204. For example, perspective cameras 202 and 204 can capture images of known objects or scenes so that differences between captured images and expected images can be identified, and such differences are used to generate a model of how perspective cameras 202 and 204 distort known objects or scenes. Imaging sensor calibration parameters represent specific parameters of the design or operation of perspective cameras 202 and 204, such as the focal length of the imaging sensor. Dedistortion and correction operations 218 and 220 can also correct or align the captured images. Image dedistortion and correction operations 218 and 220 can use any suitable technique to dedistort and correct the images. For example, image dedistortion and correction operations 218 and 220 can use intrinsic and extrinsic parameters of perspective cameras 202 and 204 to dedistort and correct the images.

[0060] The first virtual frame generation operation 222 and the second virtual frame generation operation 224 are performed by the electronic device 101 to generate the first virtual frame and the second virtual frame, respectively. For example, the first virtual frame and the second virtual frame can be determined by applying the first mapping and the second mapping (generated by the construction camera mapping operation 210) to the first image and the second image (generated by the first image capture operation 214 and the second image capture operation 216 or corrected and dedistorted by the dedistortion and correction operations 218 and 220). The first mapping can be used with the first image to generate the first virtual image, and the second mapping can be used with the second image to generate the second virtual image. The electronic device 101 can also use the overlap area map from the construction overlap area map operation 212 to perform the first overlap area compensation operation 226 and the second overlap area compensation operation 228 on the first virtual image and the second virtual image from the first virtual frame generation operation 222 and the second virtual frame generation operation 224, respectively. For example, the overlap area map can be used to compensate for artifacts or holes generated in viewpoint matching during the generation of virtual images.

[0061] A first geometric distortion calibration and chromatic aberration correction operation 230 and a second geometric distortion calibration and chromatic aberration correction operation 232 may be performed to identify and compensate for lens distortion or other distortions of the VST AR headset. For example, correction operations 230 and 232 may be used to identify any geometric distortions in a distorted image and remap the source image to an undistorted image as if it were captured with a perspective lens. Correction operations 230 and 232 may also calibrate the image based on known different refractive indices on the lens, which may increase false colors in the image as the distance from the center of the image increases. In some cases, chromatic aberration may be corrected based on lens data and focal length information using the configuration of the VST AR headset, wherein the electronic device 101 uses information from correction operations 230 and 232 to correct the first image and the second image based on characteristics of one or more display panels.

[0062] The first display rendering operation 234 and the second display rendering operation 236 can render the first virtual image and the second virtual image to be displayed on one or more display panels. Through the rendering operations 234 and 236, the electronic device 101 can display the first virtual image and the second virtual image on one or more display panels. According to an embodiment, the first display rendering operation 234 and the second display rendering operation 236 can be pre-rendered, or can be dynamically rendered. It should be noted that the display panel used here can represent two separate display panels (e.g., the left display panel and the right display panel that the user's eyes 244 can see respectively) or a single display panel (e.g., the user's eyes 244 can see the left and right parts of the display panel respectively).

[0063] It should be noted that Figure 2A and Figure 2B shown in or about Figure 2A and Figure 2B The described functions may be implemented in any suitable manner in the electronic device 101, the first external electronic device 102, the second external electronic device 104, the server 106 or other devices. For example, in some embodiments, Figure 2A and Figure 2B shown in or about Figure 2A and Figure 2B At least some of the described functions may be implemented or supported using one or more software applications or other software instructions executed by a processor 120 of the electronic device 101, the first external electronic device 102, the second external electronic device 104, the server 106, or other devices. In other embodiments, Figure 2A and Figure 2B shown in or about Figure 2A and Figure 2B At least some of the described functionality may be implemented or supported using dedicated hardware components. Figure 2A and Figure 2B shown in or about Figure 2A and Figure 2B The functions described may be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0064] although Figure 2A and Figure 2B An example of a process 200 for video conversion for video see-through AR is shown, but may be used for Figure 2A and Figure 2B Make various changes. For example, although Figure 2A and Figure 2B The operations in are shown as a series of operations, but Figure 2A and Figure 2B The various operations in can overlap, occur in parallel, occur in different orders, or occur any number of times. In addition, Figure 2A and Figure 2B The various components and functions in the can be combined, further subdivided, duplicated or rearranged according to specific needs. In addition, one or more additional components and functions can be included if needed or desired. In addition, although Figure 2 shows various operation pairs occurring in parallel (e.g., operations 214 to 216, operations 218 to 220, operations 222 to 224, operations 226 to 228, operations 230 to 232, or operations 234 to 236), other embodiments can perform common operations in sequence multiple times to achieve the same or similar results.

[0065] Figure 3An example arrangement 300 between the first perspective camera 202 and the second perspective camera 204 and the first virtual camera 302 and the second virtual camera 304 according to the present disclosure is shown. The arrangement 300 here represents the above Figure 2A and Figure 2B An example of the positions of the first perspective camera 202 and the second perspective camera 204 and the first virtual camera and the second virtual camera are described. In some cases, the arrangement 300 herein can be defined using the layout and configuration 206 .

[0066] like Figure 3 As shown, arrangement 300 represents the geometric construction of a direct pass-through conversion from the first perspective camera 202 and the second perspective camera 204 to the virtual cameras 302 and 304 associated with the user's eyes. The arrangement 300 here shows the relationship between the 3D scene, the perspective camera imaging, and the virtual camera imaging. The arrangement 300 also includes a depth A representative 3D object 306 in the scene 308. The 3D object 306 here is used to explain the point on the 3D object 306 307 to each of the first perspective frame 310 and the second perspective frame 312 and the first virtual frame 314 and the second virtual frame 316. The first perspective camera 202 and the second perspective camera 204 have a first focal length 318. The first perspective camera 202 generates a first perspective frame 310 (eg, a left perspective frame), and the second perspective camera 204 generates a second perspective frame 312 (eg, a right perspective frame).

[0067] The first virtual camera 302 and the second virtual camera 304 may correspond to the viewpoints of the user's eyes. The first virtual camera 302 may be located at the viewpoint of the first eye (e.g., the user's left eye), and the second virtual camera 304 may be located at the viewpoint of the second eye (e.g., the user's right eye). The pupil distance may be expressed as , which is also the second focal length 320 of the first virtual camera 302 and the second virtual camera 304. The first virtual camera 302 is associated with a first virtual frame 314 (eg, a left virtual image), and the right virtual camera 304 is associated with a second virtual frame 316 (eg, a right virtual image).

[0068] In this configuration, the first perspective camera 202 is aligned with the first virtual camera 302, and the second perspective camera 204 is aligned with the second virtual camera 304. The distance 322 between the first perspective camera 202 and the second perspective camera 204 and the first virtual camera 302 and the second virtual camera 304 is The depth 308 of the 3D object relative to the first perspective camera 202 and the second perspective camera 204 is Assuming that the first perspective camera 202 and the first virtual camera 302 can capture the same pixel in the 3D scene, the first perspective camera 202 can generate the pixel in the first perspective frame 310 The first virtual camera 302 can generate the same pixel in the first virtual frame 314. The same is true for the second perspective camera 204 and the second virtual camera 304. The mapping between the perspective camera imaging and the virtual camera imaging can be used for direct pass-through conversion in the first virtual frame generation operation 222 and the second virtual frame generation operation 224.

[0069] although Figure 3 An example of an arrangement 300 between the first perspective camera 202 and the second perspective camera 204 and the first virtual camera 302 and the second virtual camera 304 is shown, but the arrangement 300 may be different. Figure 3 Various changes may be made. For example, the spacing of the various components in arrangement 300 may be varied as needed or desired. Furthermore, arrangement 300 may be used in any other suitable video conversion process and is not limited to the specific process described above.

[0070] Figure 4A An example geometry 400 of direct pass-through with constant depth in VST AR according to the present disclosure is shown. Figure 4B An example geometry 401 of direct pass-through with constant depth in VST AR according to the present disclosure is shown.

[0071] Specifically, Figure 4A An example camera geometry 400 is shown, and Figure 4B An example imaging geometry 401 is shown. The camera geometry 400 and the imaging geometry 401 herein may represent Figure 2A An example of information from the get camera position operation 208 used by the build camera map operation 210 is shown in FIG.

[0072] like Figure 3 , Figure 4A and Figure 4B As shown, example geometric constructions 400 and 401 show the relationship between constant depth scenes, perspective camera imaging, and virtual camera imaging. A 3D scene can be projected onto a plane 402, the plane is located at a depth from the first perspective camera 202 308 and is perpendicular to the main optical axis 404 of the first perspective camera 202. Since the 3D scene is projected onto the plane 402 having a constant depth 308 relative to the first perspective camera 202, the projected scene is a constant scene. 307 Projection onto a plane 402 as pixel 406. The first perspective camera 202 and the first virtual camera 302 can both see the same pixel 406 pixels 408 For the first virtual camera 302 as a pixel 410 pixels 408 and pixels The relationship between 410 can be seen through their relationship with pixels 406 relationship to generate.

[0073] In some embodiments, the pixel 408 ( ) and pixels 406 ( ) can be defined as follows.

[0074] (1)

[0075] here, Represents pixels 406 is the distance from the main axis 404 along the x-axis, Represents pixels 408 is the distance from the main axis 404 along the x-axis, 308 represents pixels 406 is the distance from the first perspective camera 202, and 318 represents pixels 408 is a distance from the first perspective camera 202, which may be a first focal length. 410 ( ) and pixels 406 ( ) can be defined as follows.

[0076] (2)

[0077] here, Represents pixels 410 is the distance from the main axis 404 along the x-axis, 412 represents the distance between the first perspective camera 202 and the first display panel 414, 416 represents the distance between the first virtual camera 302 and the first display panel 414, and 320 represents pixels 410 is the distance from the first virtual camera 302 along the x-axis, which may be the second focal length. Formulas (1) and (2) may be combined into the following formula, where equal Formulas (1) and (2) can be combined into the following formula.

[0078] (3)

[0079] Similarly, pixels 408 ( ) and pixels 406 ( ) can be defined as follows.

[0080] (4)

[0081] here, Represents pixels 406 is the distance from the first perspective camera 202 along the y-axis, and Represents pixels 408 is the distance from the first perspective camera 202 along the y-axis. 410 ( ) and pixels 406 ( ) can be defined as follows.

[0082] (5)

[0083] here, Represents pixels 410 is the distance from the first virtual camera 302 along the y-axis, and Represents pixels 410 is the distance from the first virtual camera 302 along the y-axis. Formulas (4) and (5) can be combined into the following formula.

[0084] (6)

[0085] By combining equations (3) and (6), the first mapping can be defined as follows.

[0086] (7)

[0087] here, represents pixel 408 in first perspective frame 310, and Pixel 410 in the first virtual frame 314 is represented.

[0088] Although Figure 4A and Figure 4B An example of a direct pass-through geometry 400 and 401 with constant depth in VST AR is shown, but can be modified Figure 4A and Figure 4BVarious changes may be made. For example, the spacing of the various components of the camera geometry 400 and the imaging geometry 401 may be varied as needed or desired. In addition, equations (1) to (7) may be applied to other mappings for other camera viewpoints and matching virtual viewpoints. As a specific example, the same type of method may be used to generate a second mapping associated with the second perspective camera 204 and the second virtual camera 304. In addition, the camera geometry 400 and the imaging geometry 401 may be used in any other suitable video conversion process and are not limited to the specific process described above.

[0089] Figure 5 An example stereoscopic view geometry structure 500 of direct pass-through conversion according to the present disclosure is shown. Here, the stereoscopic view geometry structure 500 may represent Figure 2A An example of information from the get camera position operation 208 used by the build camera map operation 210 is shown in FIG. Figure 3 and Figure 5 As shown, the stereoscopic view geometry 500 shows the geometry of direct pass-through conversion in stereoscopic view. Here, the electronic device 101 can generate the first virtual frame 314 and the second virtual frame 316 from the first perspective frame 310 and the second perspective frame 312 according to the stereoscopic view geometry 500 of the perspective camera arrangement 300.

[0090] Using formula (7), the electronic device 101 can generate a first virtual frame 314 from the first perspective frame 310, and generate a second virtual frame 316 from the second perspective frame 312. For the first virtual frame 314, the electronic device 101 can generate a first virtual frame 314 from the pixels 504 Generating Pixels 502, for example by using the following method.

[0091] (8)

[0092] here, represents the coordinates in the first perspective frame 310, and represents the corresponding coordinates in the first virtual frame 314 .

[0093] For the second virtual frame 316, the electronic device 101 can 508 generated pixels 506, for example by using the following method.

[0094] (9)

[0095] here, represents the coordinates in the second perspective frame 312, and represents the corresponding coordinates in the second virtual frame 316 .

[0096] In some embodiments, mappings can be created using lookup tables for the first perspective camera 202 and the second perspective camera 204 and the first virtual camera 302 and the second virtual camera 304 using equations (8) and (9). The lookup tables can then be accessed to quickly use these mappings. However, it should be noted that the mappings can be used in any other suitable manner.

[0097] although Figure 5 An example of a stereoscopic view geometry 500 for direct pass-through transformation is shown, but Figure 5 Various modifications may be made. For example, the spacing of the various components of the stereoscopic view geometry 500 may be varied as needed or desired. Furthermore, the stereoscopic view geometry 500 may be used in any other suitable video conversion process and is not limited to the specific process described above.

[0098] Figure 6 6 shows an example overlap compensation 600 between a first virtual frame 314 and a second virtual frame 316 according to the present disclosure. The overlap compensation 600 here represents how it can be performed. Figure 2A An example of construct overlapping area map operation 212 is shown in FIG.

[0099] like Figure 6 As shown, overlap compensation 600 shows the layout of the first perspective camera 202 and the second perspective camera 204 and the first virtual camera 302 and the second virtual camera 304 in 3D space. The first perspective camera 202 is aligned with the first virtual camera 302, and the second perspective camera 204 is aligned with the second virtual camera 304. There is a perspective overlap area 602 between the first perspective frame 310 and the second perspective frame 312. Similarly, there is a virtual overlap area 604 between the first virtual frame 314 and the second virtual frame 316.

[0100] In some embodiments, a third mapping may be generated between the virtual overlap region 604 and the first virtual frame 314, a fourth mapping may be generated between the virtual overlap region 604 and the second virtual frame 316, and a fifth mapping may be generated between the two overlap regions 602 and 604. In certain embodiments, the third mapping may refer to a combination of the third mapping and the fifth mapping, and the fourth mapping may refer to a combination of the fourth mapping and the fifth mapping. The third mapping may be used to identify missing information in the first virtual frame 314 due to occlusion, and the fourth mapping may be used to identify missing information in the second virtual frame 316 due to occlusion.

[0101] For example, for a pixel in projection plane 402 606, is the projected pixel 608 in the first perspective frame 310, and is the corresponding projected pixel 610 in the second perspective frame 312. Both projected pixels 608 and 610 are located in the perspective overlap region 602. Pixel 608 and pixels 610 overlap in the perspective overlap region 602. Similarly, pixels 612 and pixels 614 are pixels 608 and pixels 610 is the projection of the first virtual frame 314 and the second virtual frame 316. It is possible to determine the pixel 612 and pixels 614 and pixels 608 and pixels The relationship between 610 and pixels 614 and the first virtual frame 314, and the relationship between the pixels 612 and the second virtual frame 316. Through these mappings, the electronic device 101 can use the information in the viewpoint overlap region 604 between the first virtual frame 314 and the second virtual frame 316 to compensate for any holes that may be generated due to operations such as viewpoint matching or parallax correction.

[0102] Although Figure 6 An example of overlap compensation 600 between the first virtual frame 314 and the second virtual frame 316 is shown, but may be Figure 6 Various modifications may be made. For example, the spacing of the various components of overlap compensation 600 may be varied as needed or desired. Furthermore, overlap compensation 600 may be used in any other suitable video conversion process and is not limited to the specific process described above.

[0103] Figure 7 An example coordinate relationship 700 in the overlapping region between the first virtual frame 314 and the second virtual frame 316 according to the present disclosure is shown. The coordinate relationship 700 here represents how to determine Figure 2A An example of construct overlapping area map operation 212 is shown in FIG.

[0104] like Figure 6 and Figure 7As shown, the coordinate relationship 700 shows the coordinate relationship in the virtual overlap region 604 between the first virtual frame 314 and the second virtual frame 316. The electronic device 101 can use the pixel 612 in the virtual overlap region 604 of the first virtual frame 314 to replace the corresponding pixel 614 of the second virtual frame 316, and the second virtual frame 316 has a hole at the pixel 614. In addition, the electronic device 101 can use the pixel 614 in the virtual overlap region 604 of the second virtual frame 316 to replace the corresponding pixel 612 of the first virtual frame 314, and the first virtual frame 314 has a hole at the pixel 612.

[0105] As an example, suppose the pixel 702 is a pixel on the first perspective frame 310, and the pixel 704 is a pixel on the second perspective frame 312. 702 and pixels 704 may overlap each other in the perspective overlap region 602. Two overlap points may be generated in the overlap region of the first virtual frame 314 and the second virtual frame 316. 706 and 708, such as by using equations (8) and (9) above. A third mapping between the overlapping regions created by pixels in the first virtual frame 314 and the second virtual frame 316 may be generated as follows.

[0106] (10) ,

[0107] Furthermore, a fourth mapping between the overlapping regions created by pixels in the second virtual frame 316 and the first virtual frame 314 may be generated as follows.

[0108] (11) ,

[0109] It may be an operation of searching for a point corresponding to a hole pixel having missing image information in two overlapping points and replacing the hole pixel with the searched point. Through the mapping in formula (10), the hole pixel of the first virtual frame 314 in the overlapping area can be replaced with the pixel of the second virtual frame 316. Similarly, through the mapping in formula (11), the hole pixel of the second virtual frame 316 in the overlapping area can be replaced with the pixel of the first virtual frame 314. Using this compensation, when at least one camera can capture information associated with the hole area, the hole effect can be reduced or eliminated.

[0110] In some embodiments, the distance between the first virtual camera 302 and the second virtual camera 304 may be based on the pupil distance. In these embodiments, equations (10) and (11) may be reduced, for example, in the following manner.

[0111] (12)

[0112] (13)

[0113] here, Represents the distance between the first virtual camera 302 and the second virtual camera 304 (it may also represent the pupil distance of the user's eyes).

[0114] Although Figure 7 An example of the coordinate relationship 700 in the overlapping area between virtual camera frames is shown, but the Figure 7 Various modifications may be made. For example, the spacing of the various components of coordinate relationship 700 may be varied as needed or desired. In addition, coordinate relationship 700 may be used in any other suitable video conversion process and is not limited to the specific process described above.

[0115] Fig. 8A Example compensation of unoccluded areas in a stereo image pair according to the present disclosure is shown. Figure 8B Example compensation of dis-occluded areas in a stereoscopic image pair according to the present disclosure is shown.

[0116] like Fig. 8A and Figure 8B As shown, the scene includes three objects, namely a toy mailbox, a Deathwing figure, and a Pikachu figure. In the first stereoscopic image 800, a portion of the Pikachu figure in the circled area is blocked by the Deathwing figure. In the second image 802, the blocked portion of the Pikachu figure in the first image 800 can be seen in the circled area. In the viewpoint matching conversion of the first image 800, the information from the second image 802 can be used to eliminate the blocked portion of the Pikachu figure. Similarly, the missing information in the second image 802 can be found in the first image 800 in the overlapping area of ​​the pair. Fig. 8A and Figure 8B The box in represents the overlapping area of ​​the first image 800 and the second image 802.

[0117] Although Fig. 8A and Figure 8B An example of compensation for unoccluded areas in a pair of stereo images 800 and 802 is shown, but Fig. 8A and Figure 8B Various changes may be made. For example, the number and arrangement of objects in a pair of stereo images 800 and 802 may be varied as needed or desired. Furthermore, the example compensation may be used in any other suitable video conversion process and is not limited to the specific process described above.

[0118] Fig. 9A Example deocclusion based on moving the camera away from an object in accordance with the present disclosure is shown. Fig. 9B Example deocclusion based on moving the camera away from an object in accordance with the present disclosure is shown. Fig. 9C Example deocclusion based on moving the camera away from an object in accordance with the present disclosure is shown.

[0119] Here, deocclusion indicates how it can be performed Figure 2B An example of a first overlapping area compensation operation 226 and a second overlapping area compensation operation 228 is shown in FIG.

[0120] like 9A to 9C As shown, when the camera is moved away from the object in the depth direction, the occluded area can become unoccluded. As shown here, the first image 900 can be captured at a first viewpoint 902 with a first depth 904, and the second image 906 can be captured at a second viewpoint 908 with a second depth 910. The second depth 910 can be greater than the first depth 904. From the first image 900, some areas of the mailbox and the Pikachu character are occluded by the Deathwing character. From the second image 906, the occluded area of ​​the mailbox in the first image 900 (e.g., the handle area) and the occluded area of ​​the Pikachu figure in the first image 900 (e.g., the eye area) become unoccluded, which means that one or more occluded areas in the first image 900 become visible in the second image 906. If the second image 906 is created by converting the first image 900 from the first viewpoint 902 to the second viewpoint 908, the occluded area becomes a hole after the conversion because the first image 900 does not have any information in the occluded area of ​​the occluded object. The missing information may be estimated by acquiring information from surrounding pixels, such as by using information from previous image captures, or generated in any other suitable manner to restore or fill in the missing information. According to embodiments of the present disclosure, the holes may be replaced with information retrieved from the stereo images.

[0121] although 9A to 9C shows an example deocclusion based on moving the camera away from the object, but can be 9A to 9C Various modifications may be made. For example, the number and arrangement of objects in the deocclusion may vary as needed or desired. Furthermore, the deocclusion may be used with any other suitable video conversion process and is not limited to the specific process described above.

[0122] Fig.10 An example method 1000 for direct pass-through in a VST AR according to the present disclosure is shown. For ease of explanation, Fig.10 The method 1000 is described as using Figure 1The method 1000 is performed by the electronic device 101 in the network configuration 100. However, the method 1000 may be performed by any other suitable device and any other suitable system.

[0123] like Figure 3 and Fig.10 As shown, at step 1002, the electronic device 101 receives a first image from a first camera viewpoint and receives a second image from a second camera viewpoint. For example, the first image may be received from a first perspective camera 202 having a first camera viewpoint, and the second image may be received from a second perspective camera 204 having a second camera viewpoint. The first viewpoint and the second viewpoint may correspond to a left viewpoint and a right viewpoint, and the first perspective camera 202 and the second perspective camera 204 may correspond to a left perspective camera and a right perspective camera. The first image and the second image may capture a scene from different viewpoints separated by a distance between the first camera and the second camera. The arrangement of the first perspective camera 202 and the second perspective camera 204 and the first camera viewpoint and the second camera viewpoint may be fixed and known by the electronic device 101.

[0124] At step 1004, the electronic device 101 generates a first virtual image by applying a first mapping between a first virtual camera viewpoint and a first virtual viewpoint. A first virtual image corresponding to the first virtual viewpoint may be generated by applying the first mapping to the first image. The first mapping may be based on the relative position of the first camera viewpoint and the first virtual viewpoint corresponding to the first eye of the user. In some cases, the first mapping may be defined by formula (8). At step 1006, the electronic device 101 generates a second virtual image by applying a second mapping between a second virtual camera viewpoint and a second virtual viewpoint. A second virtual image corresponding to the second virtual viewpoint may be generated by applying the second mapping to the second image. The second mapping may be based on the relative position of the second camera viewpoint and the second virtual viewpoint corresponding to the second eye of the user. In some cases, the second mapping may be defined by formula (9).

[0125] At step 1008, the electronic device 101 determines that the virtual image has missing information for at least one pixel within the first camera viewpoint and the second camera viewpoint. In some cases, the missing information for the pixel may be caused by a conversion from a perspective frame to a virtual frame. In some embodiments, the electronic device 101 may locate pixels in the virtual frame that do not have information provided from the corresponding perspective frame. At step 1010, the electronic device 101 applies an overlapping area mapping to retrieve missing image information from corresponding pixels in the relative virtual image. For example, a third mapping may be applied between the first virtual image and the second virtual image to retrieve missing image information from corresponding pixels in the second virtual image. The third mapping may be based on the pupil distance between the first virtual viewpoint and the second virtual viewpoint. Additionally or alternatively, a fourth mapping may be applied between the first virtual image and the second virtual image to retrieve missing image information from corresponding pixels in the first virtual image. The fourth mapping may be based on the pupil distance between the first virtual viewpoint and the second virtual viewpoint. In some cases, the third mapping and the fourth mapping may be defined in formulas (12) and (13), respectively.

[0126] At step 1012, the electronic device 101 presents a first virtual image and a second virtual image. For example, the first virtual image may be presented to a first virtual viewpoint, and the second virtual image may be presented to a second virtual viewpoint. The first virtual viewpoint and the second virtual viewpoint may be associated with at least one display panel of the AR device. For example, the first virtual image may be presented on a first display panel or a first portion of a common display panel, and the second virtual image may be presented on a second display panel or a second portion of a common display panel.

[0127] although Fig.10 An example of a method 1000 for direct transparent transmission in VST AR is shown, but it can be used for Fig.10 Make various changes. For example, although Fig.10 The steps in are shown as a series of steps, but Fig.10 The steps in can overlap, occur in parallel, occur in a different order, or occur any number of times.

[0128] Although the present disclosure has been described by way of example embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: obtaining a first image from a first perspective camera having a first camera viewpoint and obtaining a second image from a second perspective camera having a second camera viewpoint; generating a first virtual image corresponding to a first virtual viewpoint by applying a first mapping to the first image, the first mapping being based on a relative position of the first camera viewpoint and the first virtual viewpoint corresponding to a first eye of a user; generating a second virtual image corresponding to a second virtual viewpoint by applying a second mapping to the second image, the second mapping being based on a relative position of the second camera viewpoint and the second virtual viewpoint corresponding to a second eye of the user; as well as The first virtual image is displayed to the first virtual viewpoint on at least one display panel of an augmented reality (AR) device, and the second virtual image is displayed to the second virtual viewpoint.

2. The method according to claim 1, wherein: The first mapping is defined as: in, is the pixel position in the first image, is the corresponding pixel position in the first virtual image, is the distance between the first virtual viewpoint and the at least one display panel, is the distance between the first camera viewpoint and the at least one display panel, and is the constant depth of the projection plane to the first perspective camera.

3. The method according to claim 1, wherein: The second mapping is defined as: in, is the pixel position in the second image, is the corresponding pixel position in the second virtual image, is the distance between the second virtual viewpoint and the at least one display panel, is the distance between the second camera viewpoint and the at least one display panel, and is the constant depth of the projection plane to the second perspective camera.

4. The method according to claim 1, further comprising: Determining that the first virtual image has missing image information for pixels in the first camera viewpoint and the second camera viewpoint; as well as The missing image information is retrieved from corresponding pixels in the second virtual image by applying a third mapping between the first virtual image and the second virtual image, the third mapping being based on a pupil distance between the first virtual viewpoint and the second virtual viewpoint.

5. The method according to claim 4, wherein: The third mapping is defined as: in, is a pixel of the first virtual image, is a pixel of the second virtual image, and is the pupil distance between the first virtual viewpoint and the second virtual viewpoint.

6. The method according to claim 1, further comprising: Determining that the second virtual image has missing image information for pixels within the first camera viewpoint and the second camera viewpoint; as well as The missing image information is retrieved from corresponding pixels in the first virtual image by applying a fourth mapping between the first virtual image and the second virtual image, the fourth mapping being based on a pupil distance between the first virtual viewpoint and the second virtual viewpoint.

7. The method according to claim 6, wherein: The fourth mapping is defined as: in, is a pixel of the first virtual image, is a pixel of the second virtual image, and is the pupil distance between the first virtual viewpoint and the second virtual viewpoint.

8. A video perspective augmented reality (VST AR) device, comprising: at least one display panel; a first perspective camera and a second perspective camera; as well as At least one processing device is configured to: obtaining a first image from the first perspective camera having a first camera viewpoint and obtaining a second image from the second perspective camera having a second camera viewpoint; generating a first virtual image corresponding to a first virtual viewpoint by applying a first mapping to the first image, the first mapping being based on a relative position of the first camera viewpoint and the first virtual viewpoint corresponding to a first eye of a user; generating a second virtual image corresponding to a second virtual viewpoint by applying a second mapping to the second image, the second mapping being based on a relative position of the second camera viewpoint and the second virtual viewpoint corresponding to a second eye of the user; as well as The first virtual image is displayed to the first virtual viewpoint, and the second virtual image is displayed to the second virtual viewpoint on the at least one display panel.

9. The VST AR device according to claim 8, wherein: The first mapping is defined as: in, is the pixel position in the first image, is the corresponding pixel position in the first virtual image, is the distance between the first virtual viewpoint and the at least one display panel, is the distance between the first camera viewpoint and the at least one display panel, and is the constant depth of the projection plane to the first perspective camera.

10. The VST AR device according to claim 8, wherein: The second mapping is defined as: in, is the pixel position in the second image, is the corresponding pixel position in the second virtual image, is the distance between the second virtual viewpoint and the at least one display panel, is the distance between the second camera viewpoint and the at least one display panel, and is the constant depth of the projection plane to the second perspective camera.

11. The VST AR device according to claim 8, wherein: The at least one processing device is further configured to: Determining that the first virtual image has missing image information for pixels in the first camera viewpoint and the second camera viewpoint; as well as The missing image information is retrieved from corresponding pixels in the second virtual image by applying a third mapping between the first virtual image and the second virtual image, the third mapping being based on a pupil distance between the first virtual viewpoint and the second virtual viewpoint.

12. The VST AR device according to claim 11, wherein: The third mapping is defined as: in, is a pixel of the first virtual image, is a pixel of the second virtual image, and is the pupil distance between the first virtual viewpoint and the second virtual viewpoint.

13. The VST AR device according to claim 8, wherein: The at least one processing device is further configured to: determining that the second virtual image has missing image information for pixels within the first camera viewpoint and the second camera viewpoint; and The missing image information is retrieved from corresponding pixels in the first virtual image by applying a fourth mapping between the first virtual image and the second virtual image, the fourth mapping being based on a pupil distance between the first virtual viewpoint and the second virtual viewpoint.

14. The VST AR device according to claim 13, wherein: The fourth mapping is defined as: in, is a pixel of the first virtual image, is a pixel of the second virtual image, and is the pupil distance between the first virtual viewpoint and the second virtual viewpoint.

15. A computer readable medium comprising instructions which, when executed, cause at least one processor to perform the method according to any one of claims 1 to 7.

Images