Mesh transformation with efficient depth reconstruction and filtering in transmissive augmented reality (AR) systems

Abstract

A method includes obtaining an image of an environment captured by an imaging sensor associated with a transmissive AR device and position data and depth data associated with the image. The method further includes generating a point cloud representing the environment based on the image, the location data, and the depth data. The method further includes generating a grid of the specified image. The grid includes grid points at intersections of grid lines. The method further includes determining one or more depths of one or more grid points of the grid. The method further includes transforming the grid from a viewpoint of a specified imaging sensor capturing the specified image to a user viewpoint of the transmissive AR device based on the depth of the grid points. Further, the method includes rendering a virtual view of the specified image based on the transformed mesh for presentation by the transmissive AR device.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality (AR) systems and processes. More specifically, the present disclosure relates to mesh transformation with efficient depth reconstruction and filtering in a see-through AR system. Background Art

[0002] Augmented reality (AR) systems have become increasingly popular over time, and a large number of applications for AR systems have been developed, with many more being developed. AR systems typically enhance a user's view of his or her current environment by overlaying digital content (such as information or virtual objects) on the user's view of the current environment. For example, AR systems are often able to seamlessly blend virtual objects generated by computer graphics with real-world scenes. Summary of the invention

Technical Solution

[0003] The present disclosure relates to mesh transformation with efficient depth reconstruction and filtering in a see-through augmented reality (AR) system.

[0004] In a first embodiment, a method includes obtaining an image of an environment captured by an imaging sensor associated with a penetrating AR device, location data associated with the image, and depth data associated with the image. The method also includes generating a point cloud representing the environment based on the image, the location data, and the depth data. The method also includes generating a grid of a specified image in the image, wherein the grid includes grid points at intersections of grid lines. The method also includes determining one or more depths of one or more grid points of the grid. The method also includes transforming the grid from a viewpoint of a specified imaging sensor in the imaging sensor that captures the specified image to a user viewpoint of the penetrating AR device based on the one or more depths of the one or more grid points. In addition, the method includes rendering a virtual view of the specified image based on the transformed grid for presentation by the penetrating AR device.

[0005] In a second embodiment, a penetrating AR device includes an imaging sensor configured to capture an image of an environment. The penetrating AR device also includes at least one processing device configured to obtain an image, location data associated with the image, and depth data associated with the image. The at least one processing device is also configured to generate a point cloud representing the environment based on the image, the location data, and the depth data. The at least one processing device is also configured to generate a grid of a specified image in the image, wherein the grid includes grid points at intersections of grid lines. The at least one processing device is also configured to determine one or more depths of one or more grid points of the grid. The at least one processing device is also configured to transform the grid from a viewpoint of a specified imaging sensor in the imaging sensor that captures the specified image to a user viewpoint of the penetrating AR device based on the one or more depths of the one or more grid points. In addition, the at least one processing device is configured to render a virtual view of the specified image based on the transformed grid. The penetrating AR device also includes at least one display configured to present the virtual view.

[0006] In a third embodiment, a non-transitory machine-readable medium includes instructions that, when executed, cause at least one processor of a see-through AR device to obtain an image of an environment captured by an imaging sensor associated with the see-through AR device, location data associated with the image, and depth data associated with the image. The non-transitory machine-readable medium also includes instructions that, when executed, cause at least one processor to generate a point cloud representing the environment based on the image, the location data, and the depth data. The non-transitory machine-readable medium also includes instructions that, when executed, cause at least one processor to generate a grid of a specified image in the image, wherein the grid includes grid points at intersections of grid lines. The non-transitory machine-readable medium also includes instructions that, when executed, cause at least one processor to determine one or more depths of one or more of the grid points of the grid. The non-transitory machine-readable medium also includes instructions that, when executed, cause at least one processor to transform the grid from a viewpoint of a specified imaging sensor capturing the specified image in the imaging sensor to a viewpoint of a user of the see-through AR device based on one or more depths of the one or more grid points. Additionally, the non-transitory machine-readable medium includes instructions that, when executed, cause at least one processor to render a virtual view of a specified image based on the transformed mesh for presentation by a see-through AR device.

[0007] Other technical features will be apparent to those skilled in the art from the following drawings, descriptions and claims.

[0008] Before proceeding to the following detailed description, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, include both direct and indirect communications. The terms "include" and "comprises," and their derivatives, mean including, but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means including, included within, interconnected with, containing, contained within, connected to or connected with, coupled to or coupled with, communicable with, collaborate with, intertwined, juxtaposed, proximate to, bound to or bound with, having, having the attributes of, having a relationship to or with, etc.

[0009] In addition, the various functions described below may be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and implemented in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or parts thereof suitable for implementation in appropriate computer-readable program codes. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a high-density disk (CD), a digital video disk (DVD), or any other type of memory. "Non-transient" computer-readable media excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transient computer-readable media include media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical disks or erasable memory devices.

[0010] As used herein, terms and phrases such as "having", "may have", "include", or "may include" a feature (e.g., a number, function, operation, or component such as a part) indicate the presence of the feature without excluding the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B", or "at least one of A or B" may indicate all of the following situations: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of importance and do not limit these components. These terms are only used to distinguish components from each other. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the importance or order of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.

[0011] It should be understood that when an element (such as a first element) is referred to as being (operably or communicatively) "coupled to" or "connected to" another element (such as a second element), the element may be directly coupled to or connected to the other element, or may be coupled to or connected to the other element via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being "directly coupled to" or "directly connected to" another element (such as a second element), or "directly coupled to" or "directly connected to" another element (such as a second element), no other element (such as a third element) is interposed between the element and the other element.

[0012] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for", "capable of", "designed to", "suitable for", "manufactured to", or "capable of...", depending on the scenario. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to". More specifically, the phrase "configured to" may mean that a device is able to perform an operation together with another device or component. For example, the phrase "the processor is configured (or set) to perform A, B, and C" may represent a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.

[0013] The terms and phrases used herein are provided only to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly states otherwise. All terms and phrases used herein, including technical and scientific terms and phrases, have the same meanings commonly understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It should also be understood that terms and phrases such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that in the context of the relevant technology, and should not be interpreted as an ideal or overly formal meaning, unless explicitly defined as such herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0014] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an electronic book (e-book) reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (e.g., smart glasses, a head mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., SAMSUNG HOMESYNC, APPLE TV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (e.g., SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZONECHO), a game console (e.g., XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camera, or an electronic photo frame. Additional examples of electronic devices include at least one of the following: various medical devices (e.g., multifunctional portable medical measuring devices (e.g., blood sugar measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resonance angiography (MRA) devices, magnetic resonance imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDRs), flight data recorders (FDRs), automotive infotainment devices, marine electronic devices (e.g., marine navigation devices or gyrocompasses), avionics equipment, security devices, vehicle head units, industrial or home robots, automated teller machines (ATMs), point-of-sale (POS) devices, or Internet of Things (IoT) devices (e.g., light bulbs, various sensors, electric or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters, or kettles). Other examples of electronic devices include: at least a portion of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring instruments (e.g., devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one of the devices listed above or a combination thereof. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include any other electronic device now known or later developed.

[0015] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (eg, an artificial intelligence electronic device).

[0016] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0017] The description in this application should not be read as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims. The scope of the patented subject matter is limited only by the claims. In addition, the claims are not intended to invoke 35 USC § 112(f) unless the exact words "means for..." are followed by a participle. Applicants interpret the use of any other terms in the claims (including but not limited to "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor", or "controller") to refer to structures known to those skilled in the relevant art and are not intended to invoke 35 USC § 112(f). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description in conjunction with the accompanying drawings, in which:

[0019] Figure 1 An example network configuration including electronic devices according to the present disclosure is shown;

[0020] Figure 2 An example method for providing see-through augmented reality (AR) using mesh transformation with efficient depth reconstruction and filtering according to the present disclosure is shown;

[0021] Figure 3 An example architecture of mesh transformation with efficient depth reconstruction and filtering in a see-through AR system according to the present disclosure is shown;

[0022] Figure 4 The present invention is shown in Figure 3 Example process of performing deep filtering and reconstruction of a mesh in an architecture;

[0023] Figure 5 The present invention is shown in Figure 3 An example process for performing depth-based mesh transformation in an architecture of

[0024] Figure 6An example of mesh transformation based on color texture guided depth according to the present disclosure is shown;

[0025] Figure 7 An example of depth-based mesh transformation according to the present disclosure is shown;

[0026] Figure 8 An example of depth-based mesh transformation in a see-through AR system according to the present disclosure is shown;

[0027] Fig. 9 shows an example of a coordinate system that may be used during depth-based mesh transformation according to the present disclosure;

[0028] Fig.10 shows an example of parallax correction and occlusion overlay for a virtual object according to the present disclosure; and

[0029] Fig.11 An example of a method for mesh transformation with efficient depth reconstruction and filtering in a see-through AR system according to the present disclosure is shown. DETAILED DESCRIPTION

[0030] The following discussion will be described with reference to the accompanying drawings. Figures 1 to 11 As well as various embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements to these embodiments also belong to the scope of the present disclosure. Throughout the specification and the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0031] As described above, augmented reality (AR) systems have become increasingly popular over time, and a large number of applications have been developed for AR systems, with a large number of applications being developed. AR systems generally enhance a user's view of his or her current environment by overlaying digital content (such as information or virtual objects) on the user's view of the current environment. For example, AR systems are often able to seamlessly blend virtual objects generated by computer graphics with real-world scenes.

[0032] "Optical see-through" AR systems typically allow a user to view his or her environment directly, where light from the user's environment is passed to the user's eyes. Digital content can be superimposed on the user's view of the environment using one or more panels or other structures, where light from the user's environment passes through these panels or other structures. In contrast, "see-through" AR systems typically use a see-through camera to capture an image of the user's environment. Digital content can be fused with the captured image, and the fused image is displayed to the user for viewing. Both approaches can provide users with a rich, contextualized augmented reality experience.

[0033] In a see-through AR pipeline, a perspective camera is typically not mounted at the exact location where the user is viewing the rendered image. Therefore, the image captured by the perspective camera can be transformed from the perspective camera's viewpoint to the viewpoint of a virtual rendering camera, where the viewpoint of the virtual rendering camera represents the location of one or more displays on which the user is viewing the rendered image. Unfortunately, in a see-through AR pipeline, the perspective camera is typically a high-resolution camera (such as current 2K or 4K cameras). Transforming a high-resolution image from the perspective camera's viewpoint to the virtual rendering camera's viewpoint can be a time-consuming, resource-intensive, and computationally expensive process, which becomes even more severe as the resolution of perspective cameras continues to increase.

[0034] The present disclosure provides a technique for mesh transformation with efficient depth reconstruction and filtering in an AR system. As described in more detail below, an image of an environment captured by an imaging sensor associated with an AR device, position data associated with the image, and depth data associated with the image are obtained. A point cloud representing the environment is generated based on the image, the position data, and the depth data. For each of at least one of the images, a mesh of the image is generated, wherein the mesh includes mesh points at intersections of grid lines within the mesh. One or more depths of one or more of the mesh points of the mesh are determined, and based on the one or more depths of the one or more grid points, the mesh is transformed from a viewpoint of a specified imaging sensor of a captured image in an imaging sensor to a user viewpoint of an AR device. A virtual view of the image is rendered based on the transformed mesh for presentation by an AR device. This can be performed for any number of images in order to generate any number of virtual views for presentation by an AR device.

[0035] In this way, the described techniques support depth reconstruction and filtering of images being processed for presentation by an AR device. Typically, image processing operations such as viewpoint and parallax correction require a high-quality depth map of the image. Instead of attempting to obtain a depth map that identifies the depth of each pixel in each high-resolution image, the described techniques create a mesh for each high-resolution image and transform the mesh from the viewpoint of a perspective camera to the viewpoint of a virtual rendering camera. This approach can significantly reduce the processing and memory resources required to perform image transformations, and can significantly reduce the amount of time required to perform image transformations. In addition, these techniques can be easily integrated into the graphics rendering pipeline of a graphics processing unit (GPU) or other component, and can be seamlessly combined with rendering algorithms implemented in a GPU or other component.

[0036] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1The embodiment of the network configuration 100 shown is for illustration only. Many embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0037] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, and a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting components 120 to 180 to each other and for transmitting communications (e.g., control messages and / or data) between these components.

[0038] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In some embodiments, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processing unit (GPU). The processor 120 is capable of performing control on at least one of the other components of the electronic device 101, and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 can perform one or more functions related to mesh transformation with efficient depth reconstruction and filtering for at least one AR application.

[0039] The memory 130 may include a volatile memory and / or a non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or a program 140. The program 140 may include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0040] The kernel 141 may control or manage system resources (such as the bus 110, the processor 120, or the memory 130) for executing operations or functions implemented in other programs (such as the middleware 143, the API 145, or the application 147). The kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 may include one or more applications that perform mesh transformation with efficient depth reconstruction and filtering for at least one AR application. These functions may be performed by a single application or by multiple applications, each of which performs one or more of these functions. For example, the middleware 143 may act as a repeater to allow the API 145 or the application 147 to communicate data with the kernel 141. Multiple applications 147 may be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by assigning a priority for using the system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 is an interface that allows the application 147 to control functions provided by the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for file control, window control, image processing, or text control.

[0041] The I / O interface 150 serves as an interface that can, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0042] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hovering input by using an electronic pen or a body part of the user.

[0043] The communication interface 170 can, for example, establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals.

[0044] Wireless communication can use, for example, long term evolution (LTE), long term evolution advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM) as a cellular communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). Network 162 or 164 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network.

[0045] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101 and transform the measured or detected information into electrical signals. For example, the sensor 180 includes one or more cameras or other imaging sensors, which can be used to capture images of scenes. The sensor 180 may also include at least one depth sensor for estimating the depth within the scene being imaged. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as red, green, and blue (RGB) sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. In addition, the sensor 180 may include one or more position sensors, such as an inertial measurement unit that can include one or more accelerometers, gyroscopes, and other components. Furthermore, the sensor 180 may include a control circuit for controlling at least one sensor included therein. Any of these sensors 180 may be located within the electronic device 101 .

[0046] In some embodiments, the electronic device 101 may be a wearable device or a wearable device (such as an HMD) on which an electronic device may be installed. For example, the electronic device 101 may represent an AR wearable device such as a headset or smart glasses. In some embodiments, the first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) on which an electronic device may be installed. In these embodiments, when the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network.

[0047] The first external electronic device 102 and the second external electronic device 104 and the server 106 may each be a device of the same or different type as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a group having one or more servers. In addition, according to certain embodiments of the present disclosure, all or some operations performed on the electronic device 101 may be performed on another other electronic device or multiple other electronic devices (e.g., electronic devices 102 and 104 or server 106). In addition, according to certain embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform some functions or services, as a substitute or supplement for performing the function or service by itself, the electronic device 101 may request another device (such as electronic devices 102 and 104 or server 106) to perform at least some functions associated with the function or service. Other electronic devices (such as electronic devices 102 and 104 or server 106) are capable of performing the requested function or additional function and transmitting the result of the execution to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as is or in addition. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 for communicating with the external electronic devices 102 and 104 or the server 106 via the network 162 or 164 , but according to an embodiment of the present disclosure, the electronic device 101 may independently operate without a separate communication function.

[0048] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one operation (or function) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may perform one or more functions related to mesh transformation with efficient depth reconstruction and filtering for at least one AR application.

[0049] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any number of various components in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0050] Figure 2 An example method 200 for providing see-through AR using mesh transformation with efficient depth reconstruction and filtering according to the present disclosure is shown. For ease of illustration, Figure 2 The method 200 is described as using Figure 1 However, any other suitable device may be used and executed in any other suitable system. Figure 2 Method 200.

[0051] like Figure 2As shown, the scene being imaged includes an object 202. In this example, the object 202 represents a tree, although the object 202 can represent any other suitable real-world object. A perspective camera 204 (which can represent an imaging sensor 180 of the electronic device 101) can be used to capture images including the object 202. These images are captured from the viewpoint of the perspective camera 204, and the perspective camera 204 has a specified position relative to the object 202 in space during image capture. Using the following technology, the electronic device 101 defines a grid 206 associated with each image of the object 202 being processed. In this example, the grid 206 includes horizontal and vertical grid lines, and grid points can be defined where the horizontal and vertical grid lines intersect. The electronic device 101 operates to generate an image on a display panel 208 (which can represent a display 160 of the electronic device 101). The displayed image is focused using a display lens 210, thereby producing an image at a position associated with a virtual rendering camera 212. The resulting image can be seen by the eye 214 of the user of the electronic device 101.

[0052] As described below, the image presented on the display panel 208 can be generated by performing a transformation of the grid 206 associated with each image of the object 202 (which results in the creation of a transformed grid 216 for each image). The transformation performed here is to transform each grid 206 (which is seen from the viewpoint of the perspective camera 204) into an associated transformed grid 216 (which is seen from the viewpoint of the virtual rendering camera 212). This is achieved by performing grid transformation using a depth-based transformation method. In addition, sparse depth points or low-resolution depth maps can be used to transform each grid 206, which means that only some random sparse depth points can be used for the high-resolution image being processed. The techniques described below support depth reconstruction and filtering, which can be used to generate new depths for grid points that lack depth information. In some embodiments, the depth of each of one or more grid points associated with the image being processed can be determined based on image and depth feature information, spatial information, object posture information, and image intensity information.

[0053] Once each grid 206 is transformed, the transformed grid 216 can be applied to the image of the object 202, thereby producing a transformed object 218 that can be presented on the display panel 208. This can be achieved by transforming the grid points of each grid 206 (which is defined in the three-dimensional or "3D" space imaged by the perspective camera 204) to corresponding grid points of the transformed grid 216 (which is defined in the 3D space virtually imaged by the virtual rendering camera 212). The transformation is based on the relationship between the 3D spaces of the perspective camera 204 and the virtual rendering camera 212. In addition, the electronic device 101 can perform parallax correction and occlusion overlap to insert digital content into the rendered image generated based on the transformed grid 216. In some cases, the digital content takes the form of one or more virtual objects 220a-220b, which in this example take the form of one or more virtual animals. The electronic device 101 may determine the disparity of each virtual object 220a-220b and the position of each virtual object 220a-220b relative to one or more real objects 202 in the image, thereby providing the ability to more efficiently blend digital content into an image.

[0054] Note that the electronic device 101 can be copied Figure 2 The same type of structure shown enables the user's two eyes 214 to view the image presented on the display panel 208 of the electronic device 101. Therefore, the electronic device 101 may include multiple perspective cameras 204 for capturing images of the user's environment, and the electronic device 101 may include one or more display panels 208 for presenting rendered images to the user's eyes 214. In some cases, the rendered images can be generated for two positions of two different virtual rendering cameras 212. It should also be noted that two independent display panels 208 (such as a left display panel and a right display panel that the user's eyes 214 can view respectively) can be used here, or a single display panel 208 (such as a display panel in which the user's eyes 214 can view the left and right portions of the display panel 208 respectively) can be used.

[0055] although Figure 2 One example of a method 200 for providing see-through AR using mesh transformation with efficient depth reconstruction and filtering is shown, but may be used for Figure 2 For example, the specific object 202 and virtual objects 220a-220b shown here are only examples, and may be varied in various ways based on the environment surrounding the electronic device 101 and the digital content presented to the user.

[0056] Figure 3 An example architecture 300 for mesh transformation with efficient depth reconstruction and filtering in a see-through AR system according to the present disclosure is shown. For ease of illustration, Figure 3 The architecture 300 is described as using Figure 1 The architecture 300 is implemented by the electronic device 101 in the network configuration 100. However, any other suitable device and any other suitable system may be used to implement the architecture 300.

[0057] like Figure 3 As shown, the architecture 300 obtains an input image 302, which represents an image captured using an imaging sensor (e.g., a perspective camera 204 of the electronic device 101 or other imaging sensor 180). Each input image 302 can have any suitable resolution and size, depending on the capabilities of the corresponding imaging sensor 180. In some cases, for example, each input image 302 can have a 2K or 4K resolution. Each input image 302 can also include image data in any suitable format. In some embodiments, for example, each input image 302 includes RGB image data, which typically includes image data of three color channels (i.e., a red channel, a green channel, and a blue channel). However, each input image 302 can include image data having any other suitable resolution, form, or arrangement.

[0058] The architecture 300 also obtains position information 304 and depth information 306 associated with the input image 302. The position information 304 may be associated with the position of the perspective camera 204 or other imaging sensor 180 when the input image 302 was captured. In some cases, the position information 304 may be captured using one or more position sensors 180 (e.g., an inertial measurement unit). The depth information 306 may be associated with different depths within the input image 302 (e.g., estimated various depths or distances between the perspective camera 204 or other imaging sensor 180 and various parts of the scene captured in the input image 302). In some cases, the depth information 306 may be captured using one or more depth sensors 180, and the captured depth information 306 may include a low-resolution depth map or a set of sparse depth points. In various implementations, the resolution of the depth information 306 may be lower (and in some cases much lower) than the resolution of the input image 302.

[0059] The input image 302 and the position information 304 are provided to a pose tracking operation 308, which generally operates to estimate the pose 310 of the perspective camera 204 or other imaging sensor 180 during the capture of the input image 302. The pose tracking operation 308 also operates to generate 3D sparse points 312, which may include a set of 3D points identified in each input image 302. In some embodiments, the pose 310 of the perspective camera 204 or other imaging sensor 180 that captured the input image 302 can be represented using six degrees of freedom (6DoF), which can include three distances and three angles. The three distances generally represent the distance of the position of the imaging sensor 180 or electronic device 101 relative to a fixed position, such as the distance of the position of the imaging sensor or electronic device from the fixed position expressed in distances along three orthogonal axes. The three angles generally represent the orientation of the imaging sensor 180 or electronic device 101 around three axes, such as the pitch angle, roll angle, and yaw angle of the imaging sensor or electronic device. Furthermore, in some cases, pose 310 and 3D sparse points 312 of input image 302 may be obtained simultaneously, for example by applying simultaneous localization and mapping (SLAM) to input image 302 and position information 304. Here, simultaneous localization and mapping may be used to track the imaging sensor pose and reconstruct 3D points of the scene.

[0060] The camera pose 310, the 3D sparse points 312, and the depth information 306 are provided to a point cloud generation operation 314, which is generally used to generate a 3D point cloud representing the surrounding environment of the electronic device 101. The 3D point cloud represents a collection of points in the 3D space around the electronic device 101, and these points are generally associated with real-world objects in the 3D space around the electronic device 101. By using the camera pose 310, the 3D sparse points 312, and the depth information 306, the point cloud generation operation 314 uses a combination of: (i) depth data collected by one or more depth sensors 180 associated with the electronic device 101, and (ii) depth data determined by applying simultaneous localization and mapping to the input image 302 and the position information 304. In some cases, the point cloud generation operation 314 integrates the 3D points determined using the input image 302 and the sparse depth points or low-resolution depth determined using the depth sensor 180 into a point cloud by performing depth fusion. During the depth fusion process, the point cloud generation operation 314 may give higher priority or weight to the depth determined using the depth sensor 180 and lower priority or weight to the depth estimated from the input image 302 .

[0061] The input image 302 is also provided to a grid generation operation 316, which generally operates to define one or more grids (such as one or more grids 206) associated with one or more real-world objects captured in the input image 302. In some embodiments, the grid generation operation 316 may define each grid 206 using substantially horizontal grid lines and substantially vertical grid lines. The number of grid lines in each direction and the spacing between adjacent grid lines may be defined in any suitable manner, such as based on a default configuration, a user's custom configuration, or a dynamic definition of the grids 206 (which may or may not depend on the image content being analyzed). The size and shape of each grid 206 may also be designed to substantially or completely cover at least the associated real-world object. As previously described, the grids 206 defined herein may represent grids as viewed from the viewpoint of the perspective camera 204 or other imaging sensor 180 that captured the input image 302.

[0062] The input image 302, 3D point cloud, and mesh are provided to a mesh depth filtering and reconstruction operation 318, which generally operates to generate a higher resolution depth set for each of the defined meshes. For example, the architecture 300 can be used to perform a transformation of the mesh 206 to a transformed mesh 216 based on the depth of the mesh points of the mesh 206, wherein each mesh point of the mesh 206 is defined at the intersection of two mesh lines. Therefore, the mesh depth filtering and reconstruction operation 318 can operate to identify the depth of each mesh point of each mesh 206 provided by the mesh generation operation 316. In some cases, if the mesh point of the mesh 206 corresponds to a depth measured by the depth sensor 180, the mesh depth filtering and reconstruction operation 318 can use the measured depth. If the mesh point of the mesh 206 does not have an associated measured depth, the mesh depth filtering and reconstruction operation 318 can use other depths to estimate the depth of the mesh point. Thus, the mesh depth filtering and reconstruction operation 318 may be operable to determine the depths of the mesh points of the mesh 206 when those depths are unknown, thereby producing a higher resolution depth map or other higher resolution depth information for the input image 302. The mesh depth filtering and reconstruction operation 318 may also clarify depths at mesh points having low confidence scores by removing spurious depths and depth noise in the higher resolution depth set.

[0063] The higher resolution depth information is provided to a depth-based mesh transformation operation 320, which generally operates to transform a mesh (such as the mesh 206) associated with the input image 302 into a transformed mesh (such as the transformed mesh 216). For example, the depth-based mesh transformation operation 320 may perform these transformations based on the relationship between the perspective camera 204 or other imaging sensor 180 and the virtual rendering camera 212. As a specific example, the depth-based mesh transformation operation 320 may perform these transformations based on: (i) the relationship between the known position of one perspective camera 204 or other imaging sensor 180 and the known position of one virtual rendering camera 212, and (ii) the relationship between the known position of another perspective camera 204 or other imaging sensor 180 and the known position of another virtual rendering camera 212. This process facilitates transforming the mesh 206 (as viewed from the viewpoint of the perspective camera 204) into the transformed mesh 216 (as viewed from the viewpoint of the virtual rendering camera 212). During mesh transformation, depth-based mesh transformation operation 320 may correct for the parallax difference between the two viewpoints for real-world information captured by perspective camera 204 or other imaging sensor 180 .

[0064] The rendering operation 322 uses the transformed grid 216 to render a left virtual view 324 and a right virtual view 326 of the environment around the electronic device 101. For example, the rendering operation 322 can deform the input image 302 and use the transformed grid 216 to blend digital content (such as one or more virtual objects 220a-220b) into the input image 302. Part of the rendering process can include occlusion overlap and parallax processing. Occlusion overlap can be performed to help ensure that the digital content properly occludes one or more real-world objects or is occluded by one or more real-world objects. Parallax processing can be performed to help compensate for parallax associated with different positions of the virtual rendering camera 212. Here, the rendering operation 322 can determine the parallax of each virtual object in the input image 302 and the position of each virtual object (relative to one or more real-world objects) based on the transformed grid 216, and can render the virtual views 324, 326 to include each virtual object based on the determined parallax and the determined position. Thus, the end result here is that the left virtual view 324 and the right virtual view 326 allow the user to see an image that integrates the transformed real-world information from the perspective camera 204 or other imaging sensor 180 with one or more virtual objects that have been corrected for parallax and distortion. The use of the transformed grid 216 here can help ensure that the digital content is inserted into the input image 302 in an efficient and pleasing manner.

[0065] In this example, the left virtual view 324 and the right virtual view 326 can represent a stereoscopic image pair generated and presented to the user. Here, the stereoscopic image pair represents a 2.5-dimensional (2.5D) view of the environment around the electronic device 101. A 2.5D view generally refers to a scene view that includes some depth information, but the depth information is not necessarily provided for every position in the scene. Instead, each of the left virtual view 324 and the right virtual view 326 in the stereoscopic image pair can include multiple pixel sets, such as pixels in different image data layers or pixels in different parts of the scene. For each pixel set, a common depth (such as the depth at the grid point of the transformed grid 216) can be associated with the pixels in the pixel set. This type of method can reduce the amount of computation required and can speed up the process of generating a 2.5D view based on the input image 302. This may be particularly useful if the architecture 300 is used to sequentially process a large number of input images 302 so as to provide a stream of stereoscopic image pairs to the display panel 208 for presentation to the user.

[0066] In general, the above-described architecture 300 can be faster and more efficient than other algorithms. For example, the architecture 300 may require only fewer computing resources to determine depth information from the input image 302, and only fewer computing resources to insert digital content into the input image 302. In addition, the architecture 300 can be easily integrated into the graphics rendering pipeline of a GPU or other component, which can reduce or avoid the need to transfer data back and forth with a CPU or other component, thereby reducing latency and memory consumption. In addition, converting the input image 302 to a 2.5D view requires less computing power and fewer resources than converting the same input image 302 to a 3D view because true depth and volume reconstruction are not required. In addition, these benefits can allow the architecture 300 to be used by electronic devices with smaller processing power.

[0067] It should be noted that Figure 3 Shown or about Figure 3 The described functions may be implemented in any suitable manner in electronic devices 101, 102, 104, server 106, or other devices. For example, in some embodiments, Figure 3 Shown or about Figure 3 At least some of the described functionality may be implemented or supported using one or more software applications or other software instructions executed by a processor 120 of an electronic device 101, 102, 104, server 106, or other device. Figure 3 Shown or about Figure 3 At least some of the described functionality may be implemented or supported using dedicated hardware components. Figure 3 Shown or about Figure 3The functions described may be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0068] although Figure 3 An example of an architecture 300 for mesh transformation with efficient depth reconstruction and filtering in a see-through AR system is shown, but may be used for Figure 3 Make various changes. For example, Figure 3 The various components and functions in the embodiment can be combined, further subdivided, duplicated, omitted or rearranged according to specific needs. In addition, if necessary or desired, one or more additional components or functions can also be included.

[0069] Figure 4 The method for executing the present invention is shown Figure 3 An example process 400 for mesh depth filtering and reconstruction 318 in the architecture 300 of FIG. 400 is shown for performing Figure 3 3. An example technique for performing at least a portion of the mesh depth filtering and reconstruction operation 318 in the architecture 300 of FIG. As described above, the mesh depth filtering and reconstruction operation 318 is used to create higher resolution depth information for the mesh 206.

[0070] like Figure 4 As shown, various extraction operations 402, 404, 406, 408 are performed to obtain information for creating higher resolution depth information. For example, the image and depth feature extraction operation 402 generally operates to extract specified features from the input image 302 and the depth information defined in the 3D point cloud provided by the point cloud generation operation 314. In some cases, the extracted features of the input image 302 may include color texture information associated with the image content of the input image 302. The specified features identified here may represent features that have been identified as being related to mesh depth filtering and reconstruction functions. As a specific example, the specified features may represent features that have been learned by the machine learning model and are related to depth filtering and reconstruction. Note that at this time, the depth information defined in the 3D point cloud provided by the point cloud generation operation 314 may represent sparse depth information (such as sparse depth points or low-resolution depth), which means that the available depth does not usually cover all points within the input image 302. In some cases, the sparse depth information may be augmented, for example, by using an interpolation technique (such as nearest neighbor interpolation) to obtain an intermediate depth map for extracting depth features.

[0071] The object pose extraction operation 404 generally operates to identify the pose of an object captured in the input image 302. For example, the object pose extraction operation 404 can identify one or more objects of one or more specified types within the input image 302, and identify the pose of each object. In some cases, each pose can be expressed using six degrees of freedom (e.g., three distances and three angles as discussed above). To facilitate identification of distances, intermediate depth information generated by the image and depth feature extraction operation 402 can be used to generate object poses. The image intensity extraction operation 406 generally operates to identify the intensity of pixels or other areas within the input image 302. The intensity information can be determined using the content of the input image 302. The spatial information extraction operation 408 generally operates to identify spatial information about the input image 302, such as the location of objects and the relative locations of objects to each other or to the electronic device 101.

[0072] The depth filter construction operation 410 generally operates to process the information obtained by the extraction operations 402, 404, 406, 408 to generate a depth filter that is subsequently applied to the grid 206 associated with the object in the input image 302. Therefore, each depth filter generated here can be created based on image feature information, depth feature information, object pose information, image intensity information, spatial information, or any suitable combination thereof. Each depth filter can define how depth information in a region surrounding a location within the input image 302 can be processed to identify depth information at the location. For example, each depth filter can identify both: (i) a neighboring region around a grid point for identifying depth information for the grid point, and (ii) a manner in which depth information in the neighboring region around the grid point is used to identify depth information for the grid point.

[0073] The depth calculation operation 412 generally operates to determine the depth at a grid point within the grid 206 provided by the grid generation operation 316. Here, the depth is determined by applying a corresponding depth filter (determined by the depth filter construction operation 410) to the depth information associated with the corresponding input image 302. In some embodiments, the depth filter is applied to a particular grid point in the grid 206 only when the particular grid point lacks known depth information. If the depth information at a particular grid point in the grid 206 is known, such as when the depth sensor 180 measures the depth at the grid point, it may not be necessary to apply a depth filter to the grid point (although this may be done to generate additional data or verify the measured depth from the depth sensor 180). Note that this may be performed only for the grid points in the grid 206 provided by the grid generation operation 316 (rather than for each pixel of each input image 302), which means that the depth filter may only be applied to information at or within a neighborhood around the grid point.

[0074] The determination operation 414 may be operable to confirm whether the depth of each grid point of the grid 206 has been obtained or estimated. If not, the depth filter construction operation 410 and the depth calculation operation 412 may be repeated for additional grid points, and this may continue until a complete depth set 416 is determined for each grid 206. The complete depth set 416 may include the measured or estimated depth of each grid point within the grid 206, and the process may be repeated for each grid 206. Here, the depth filter may be constructed based on image feature information, depth feature information, object pose information, image intensity information, and / or spatial information, so that the estimated depth may have a good correspondence with the measured depth in its defined neighborhood. In addition, certain features (such as image features) may allow the estimated depth to maintain edge characteristics and may smooth noisy depth.

[0075] although Figure 4 Shown is the method for executing Figure 3 An example of a process 400 for mesh depth filtering and reconstruction 318 in the architecture 300 of FIG. 300 , but may be used for Figure 4 Make various changes. For example, Figure 4 The various components and functions in the embodiment can be combined, further subdivided, duplicated, omitted or rearranged according to specific needs. In addition, if necessary or desired, one or more additional components or functions can also be included.

[0076] In addition, although Figure 4One specific process of mesh depth filtering and reconstruction is shown, but various other methods may also be used. For example, a deep neural network or other machine learning model may be trained to generate a feature map, such as a feature map of each image captured by each of the plurality of perspective cameras 204. Another deep neural network or other machine learning model may be trained to generate a disparity map of a corrected version of the captured image, and a depth map may be generated based on the disparity map and configuration information of the perspective camera 204. The generated depth map may be used to perform viewpoint matching and disparity correction.

[0077] Figure 5 The method for executing the present invention is shown Figure 3 An example process 500 for performing a depth-based mesh transformation 320 in the architecture 300 of FIG. Figure 3 An example technique for at least a portion of the depth-based mesh transformation operation 320 in the architecture 300 of FIG. As described above, the depth-based mesh transformation operation 320 is used to transform the mesh 206 into the transformed mesh 216 based on the depth information provided by the mesh depth filtering and reconstruction operation 318.

[0078] like Figure 5 As shown, process 500 obtains and uses a relationship 502 between one or more perspective cameras 204 (such as one perspective camera for each eye of a user) and a 3D scene imaged by perspective cameras 204. Relationship 502 can be defined in any suitable manner, and is generally defined based on the configuration of perspective cameras 204. As a specific example, relationship 502 can be based at least in part on a focal length of perspective cameras 204, a spacing of multiple perspective cameras 204, and an orientation of perspective cameras 204. Process 500 also obtains and uses a relationship 504 between one or more virtual rendering cameras 212 (such as one virtual rendering camera for each eye of a user) and a 3D scene imaged by virtual rendering cameras 212. Relationship 504 can also be defined in any suitable manner, and is generally defined based on the configuration of virtual rendering cameras 212. As a specific example, relationship 504 can be based at least in part on a virtual focal length of virtual rendering cameras 212, a virtual spacing of multiple virtual rendering cameras 212, and a virtual orientation of virtual rendering cameras 212.

[0079] The relationship identification operation 506 generally operates to generate a relationship between each perspective camera 204 and its corresponding virtual rendering camera 212. For example, the relationship identification operation 506 can process the relationships 502 and 504 to mathematically determine how the viewpoint of each perspective camera 204 will need to be transformed to create the viewpoint of the corresponding virtual rendering camera 212. Once the relationship between the perspective cameras 204 and their corresponding virtual rendering cameras 212 is known, a grid point transformation operation 508 can be performed. The grid point transformation operation 508 generally operates to transform the depth at the grid point of each grid 206 to the corresponding depth at the corresponding grid point of the transformed grid 216. A determination operation 510 can be used to confirm whether the grid point transformation operation 508 has transformed all grid points of the grid 206, and if not, the grid point transformation operation 508 can be repeated for another grid point of the grid 206. The process provides a depth-based transformation for all grid points to obtain a transformed grid 216 in which grid points of the grid 206 (associated with the viewpoint of the associated perspective camera 204) have been transformed to corresponding grid points of the transformed grid 216 (associated with the viewpoint of the associated virtual rendering camera 212). As described below, during the depth-based transformation, parallax differences between the grid 206 at the viewpoint of the perspective camera 204 and the transformed grid 216 at the viewpoint of the virtual rendering camera 212 may be corrected.

[0080] although Figure 5 Shown is the method for executing Figure 3 The process 500 of the depth-based mesh transformation 320 of the architecture 300 is an example of Figure 5 Make various changes. For example, Figure 5 The various components and functions in the embodiment can be combined, further subdivided, duplicated, omitted or rearranged according to specific needs. In addition, if necessary or desired, one or more additional components or functions can also be included.

[0081] Figure 6 An example of a mesh transformation based on color texture guided depth according to the present disclosure is shown. More specifically, Figure 6 An example of determining a depth value based on a depth filter is shown, which can be implemented using the above-described grid depth filtering and reconstruction operation 318. In this example, the grid 206 is shown to be associated with depth values ​​arranged in cells 602a-602b. Cell 602a represents a pixel whose depth value is known (such as by measuring the depth value using the depth sensor 180), and cell 602b represents a pixel whose depth value is unknown. Here, cell 604 represents a pixel at a grid point of the grid 206.

[0082] In an actual penetrating AR pipeline, only sparse depth points or low-resolution depth as described above may be obtained. If the depth of cell 604 does not exist, the depth of cell 604 can be estimated using the depth contained in the defined neighborhood 606 around cell 604. As described above, cell 604 can represent a grid point within the grid 206 defined by the grid generation operation 316, and the depth at the grid point may be known or unknown. When the depth at the grid point is unknown, the grid depth filtering and reconstruction operation 318 can generate a depth filter that defines a neighborhood 606 around the cell 604 representing the grid point. Any measured or estimated depth within the defined neighborhood 606 can be used to estimate the depth of cell 604.

[0083] In some cases, the depth filter used to define neighborhood 606 can be created based on image feature information, depth feature information, object pose information, image intensity information, spatial information, or any suitable combination thereof. Thus, image feature information, depth feature information, object pose information, image intensity information, and / or spatial information can be applied to generate a new depth from any neighborhood depth. In some embodiments, the depth filter used herein can be based on the depth in neighborhood 606. To generate a new depth In a particular embodiment, this may be expressed as follows.

[0084] (1)

[0085] here, represents the depth at the considered grid point (element 604), and represents the depth of a point in the neighborhood 606 of the considered grid point (in, ).also, represents the grid points considered The neighborhood of 606, and In this example, the weight It is defined based on image intensity information, image and depth feature information, spatial information, and available sparse depth information.

[0086] As shown in equation (1), Indicate point The depth of the points in the neighborhood 606 of . Since only sparse depth information is available for input to the penetrating AR pipeline, the depth in the neighborhood 606 can be used to estimate the dense depth map. In some cases, the depth of the grid points whose depth is unknown can be calculated using nearest neighbor interpolation. In some cases, this can be expressed as follows.

[0087] (2)

[0088] here, Represents points determined using the nearest neighbor interpolation method The estimated depth at represents the nearest neighbor interpolation method, and Indicate point Neighborhood 606.

[0089] As shown in equation (1), Denote weights, which may be based on image intensity information, image and depth feature information, spatial information, and available sparse depth information. In some cases, weights may be defined as follows.

[0090] (3)

[0092] Here, item to take into account spatial information, and represents the smoothing parameter. Used to consider image intensity information, where Represents pixels The strength of is the smoothing parameter. It is used to consider the existing sparse depth information, where Indicates pixel The estimated depth of is the smoothing parameter. Used to consider image feature information, where Indicates pixel feature information, and is the smoothing parameter.

[0093] By using this method, an estimated depth for grid points within each grid 206 that do not have a known or measured depth can be determined. Since the estimated depth can be generated by considering spatial, image, and depth information, the method can provide high-quality depth values ​​for any number of grids 206. If there is a known or measured depth associated with a grid point in the grid 206, the method can still be used, for example, to estimate the depth at the grid point in order to verify whether the known or measured depth appears to be a true depth or a noisy depth. In this type of verification process, the above method can be used to estimate the depth of the grid point, and then the known or measured depth can be compared to the estimated depth. If the difference between the two is too large, the known or measured depth can be considered unreliable and can be replaced or otherwise modified using the estimated depth, thereby providing depth filtering.

[0094] although Figure 6An example of a color texture guided depth transformation for the mesh 206 is shown, but may be performed for Figure 6 Various changes may be made. For example, neighborhood 606 may be defined to include any suitable number of cells in any suitable arrangement. Furthermore, cells 602a-602b with depth values ​​and without depth values ​​may be easily varied depending on the circumstances.

[0095] Figure 7 An example of depth-based mesh transformation according to the present disclosure is shown. More specifically, Figure 7 An example of a depth-based transformation of the grid 206 to the transformed grid 216 is shown, which can be implemented using the depth-based grid transformation operation 320 described above. In this example, the grid 206 includes horizontal and vertical grid lines 702, and grid points 704 are defined where the grid lines 702 intersect. Each grid point 704 can be associated with two coordinates (such as an x-coordinate and a y-coordinate) and a depth d. The depth can represent a known or measured depth (such as a depth measured using a depth sensor 180) or an estimated depth (such as a depth estimated using a grid depth filtering and reconstruction operation 318).

[0096] Above Figure 5 The process 500 operates to transform the grid 206 into a transformed grid 216. In this example, the transformed grid 216 includes modified grid lines 702' and grid points 704' defined at the intersections of the modified grid lines 702'. Each grid line 702 has a corresponding modified grid line 702' (which may be different from or the same as the grid line 702), and each grid point 704 has a corresponding grid point 704'. Figure 5 Process 500 of transforming grid 206 to transformed grid 216 involves converting the depth at grid point 704 to the corresponding depth at grid point 704'. In some cases, this can be mathematically represented as follows.

[0097] (4)

[0098] here, represents a grid point 704', represents a grid point 704, and represents a transformation function that transforms the grid point 704 from the viewpoint of the perspective camera 204 to the viewpoint of the virtual rendering camera 212. In this manner, the original grid 206 may be transformed into the transformed grid 216. In some embodiments, OpenGL may be used to interpolate certain grid points 704' within the transformed grid 216 based on other grid points 704' within the transformed grid 216.

[0099] although Figure 7 An example of depth-based mesh transformation is shown, but Figure 7 Various changes may be made. For example, each grid 206, 216 may include any appropriate number of grid lines and any appropriate number of grid points. In some cases, the number of grid lines and grid points may be adjusted, such as according to the needs of a specific penetrating AR pipeline or a specific application in a penetrating AR device. In addition, the shapes of the grids 206, 216 shown here are only examples and can be easily changed as needed or desired.

[0100] Figure 8 An example of depth-based mesh transformation in a penetrating AR system according to the present disclosure is shown. More specifically, Figure 8 shows how the electronic device 101 or other see-through AR system can be used Figure 7 The method shown and the combination of two perspective cameras 204a-204b and two virtual rendering cameras 212a-212b. In this example, the electronic device 101 is imaging a 3D object 802, which in this example represents a tree. Assume 802 in the scene. Perspective cameras 204a-204b may be used to capture images 804 and 806 of object 802. Perspective cameras 204a-204b have camera poses 804 and 806, respectively. and ,and and Respectively represent the image points on the left perspective camera 204a and the right perspective camera 204b .value denotes the rotation and translation between the left perspective camera 204a and the left virtual rendering camera 212a, respectively, and the values represent the rotation and translation between the right perspective camera 204b and the right virtual rendering camera 212b, respectively.

[0101] The electronic device 101 operates to closely match the viewpoint of the left perspective camera 204a with the viewpoint of the left virtual rendering camera 212a, and to closely match the viewpoint of the right perspective camera 204b with the viewpoint of the right virtual rendering camera 212b. This converts the captured images 804 and 806 into overlapping virtual images 808 and 810. The virtual rendering cameras 212a-212b have camera poses and ,and and Respectively represent the image points on the left virtual rendering camera 212a and the right virtual rendering camera 212b An interpupillary distance (IPD) 812 is shown as the distance between the virtually rendered cameras 212a - 212b . Figure 8 The zero parallax point 814 in represents the position along the central optical axis where the sign of the parallax changes. For example, the parallax from the user's eye to the zero parallax point 814 may be negative, while the parallax from the human eye to the area beyond the zero parallax point 814 may be positive. The actual position of the zero parallax point 814 may vary based on the size of the pupil distance 812 (etc.).

[0102] Taking the left perspective camera 204a as an example, point and image points The relationship between can be expressed as follows.

[0103] (5)

[0104] in, represents the projection matrix of the left perspective camera 204a. In some cases, the projection matrix can be represented as follows.

[0105] (6)

[0106] here, The camera matrix representing the left perspective 204a, which in some cases may be defined as follows.

[0107] (7)

[0108] Therefore, equation (5) can be rewritten as follows.

[0109] (8)

[0110] In view of this, the image point in the coordinate system of the left perspective camera 204a may be defined as follows.

[0111] (9)

[0112] Similarly, for the left virtual rendering camera 212a, the following equation can be obtained.

[0113] (10)

[0114] (11)

[0115] By combining equation (9) and equation (11), the following equation can be obtained.

[0116] (12)

[0117] Similar equations may be derived for the right perspective camera 204b and the right virtual rendering camera 212b.

[0118] Fig. 9An example coordinate system that can be used during depth-based mesh transformation according to the present disclosure is shown. In this example, the coordinate system is relative to Figure 8 The object 802 shown is schematically illustrated. The coordinate system here includes the world coordinate system 902, which is defined by three orthogonal axes X, Y, and Z. The coordinate system here also includes the camera coordinate system 904, which is associated with perspective cameras 204a, 204b and consists of three orthogonal axes x c ,y c and z c Definition: The image 804 or 806 captured by the perspective camera 204a or 204b and the image coordinate system 906 (which is defined by two orthogonal axes x and y). The image 804 or 806 captured by the perspective camera 204a or 204b is also associated with the pixel coordinate system 908 (which is defined by two orthogonal axes u and v). In this example, the pixel coordinate system The origin of 908 is at the upper left corner of image 804 or 806 .

[0119] In some embodiments, based on the above equation (12), the posture of the perspective camera 204a can be The image 804 below is transformed into the pose of the virtual rendering camera 212a In some cases, this can be done by converting the pixel coordinate system The coordinates in 908 are converted into coordinates in a normalized device coordinate (NDC) system to obtain NDC coordinates. As a specific example, the coordinate conversion can be performed as follows.

[0120] (13)

[0121] here, represents the pixels of perspective camera 204a The depth of Represents the image width and image height respectively. In addition, x NDC ,y NDC and z NDC Represents coordinates in the NDC system. A homogeneous coordinate system can be created based on the NDC coordinates. For example, the homogeneous coordinates can be represented as follows.

[0122] (14)

[0123] Here we can see that . Then equation (12) can be rewritten as follows.

[0124] (15)

[0125] Thus, the pose of the virtual rendering camera 212a can be determined The transformed image coordinates under The pixel coordinate system can also be determined, for example, as follows Coordinates in 908.

[0126] (16)

[0127] here, Represents the virtual camera pose The pixel coordinate system The coordinates in Represents the perspective camera pose To the virtual camera pose The transformed image coordinates of Likewise, similar equations may be derived for the right perspective camera 204b and the right virtual rendering camera 212b.

[0128] although Figure 8 An example of depth-based mesh transformation in a see-through AR system is shown, but can be used for Figure 8 For example, the specific arrangement of perspective cameras 204a-204b and virtual rendering cameras 212a-212b is for illustration and description only. Fig. 9 shows an example of a coordinate system that can be used during depth-based mesh transformation, but Fig. 9 Various changes may be made. For example, each coordinate system may be defined in any other suitable manner.

[0129] In addition, although Figure 8 and Fig. 9A specific method for viewpoint matching of a perspective camera and a virtual rendering camera is supported, but various other methods may also be used. For example, the viewpoint matching method may be extended to the case with one perspective camera and the case with more than two perspective cameras. In the case with one perspective camera 204, the viewpoint of the perspective camera 204 may be matched with the viewpoint of the left virtual rendering camera 212a using the same process described above, and additionally matched with the viewpoint of the right virtual rendering camera 212b. In these examples, the one perspective camera 204 may be used as both the left perspective camera 204a and the right perspective camera 204b. In the case with more than two perspective cameras, multiple groups of perspective cameras may be defined, such as a group of perspective cameras closer to the left virtual rendering camera 212a and another group of perspective cameras closer to the right virtual rendering camera 212b. The viewpoint of the camera in the left group may be matched with the viewpoint of the left virtual rendering camera 212a using the same process described above, and the viewpoint of the camera in the right group may be matched with the viewpoint of the right virtual rendering camera 212b. During viewpoint matching, contributions of different perspective cameras in each group to the corresponding virtual view may be fused, which may allow the perspective cameras in each group to compensate each other when contributing to the virtual view.

[0130] Fig.10 An example of parallax correction and occlusion overlay 1000 for a virtual object according to the present disclosure is shown. The example shown here represents a method for performing Figure 3 300 of the architecture 300. As described above, the rendering operations 322 may include: occlusion overlap to help ensure that the digital content properly occludes or is occluded by one or more real-world objects; and parallax processing to help compensate for the parallax associated with the different positions of the virtual rendering cameras 212a-212b. In an ideal case, this can help virtual objects have the same parallax as objects in the real-world scene being imaged.

[0131] Through the depth-based mesh transformation, the mesh 206 associated with the object in the image captured by the perspective cameras 204a-204b is transformed into the mesh 216 associated with the virtual rendering cameras 212a-212b. The process also involves using the depth values ​​of the mesh points of the mesh 206 that have been estimated as described above to generate higher resolution depth values. The use of higher resolution depth values ​​enables the generation of the transformed mesh 216 and the performance of parallax correction and occlusion overlap for virtual objects (such as virtual objects generated by the graphics pipeline).

[0132] For parallax correction involving one or more virtual objects, one or more transformed meshes 216 may be used to render the one or more virtual objects so that the parallax of each virtual object is the same or similar to the parallax in the real world scene. This is possible because the depth used is also obtained from the real world scene. For example, in Fig.10 In FIG. 1 , the real world scene includes two real world objects 1002 and 1004, which in this example represent trees. In this example, two virtual objects 1006 and 1008 are inserted into the real world scene, and the virtual objects 1006 and 1008 represent two animals (tiger and horse). Fig.10 As shown, virtual objects 1006 and 1008 may be rendered at appropriate sizes and positions based on the depth information generated using the above-described method.

[0133] For occlusion overlaps involving one or more virtual objects, one or more transformed grids 216 may be used to determine which virtual object(s) appear in front of one or more real-world objects and which virtual objects are at least partially occluded by one or more real-world objects. Fig.10 , virtual object 1006 is rendered in front of real-world object 1002, while virtual object 1008 is rendered behind real-world object 1002 (and therefore partially obscured by real-world object 1002).

[0134] As described above, the color texture guided depth generation method determines image features for depth value generation, which may include object outlines. The depth on the object outline (e.g., the outline of each real-world object 1002 and 1004) can also be determined. Using the depth determined on the object outline, it can be determined which part or parts (if any) of each virtual object 1006, 1008 is occluded. Depth values ​​can also be generated in selected areas and for specific object features so that these depth values ​​can be used for different purposes. In addition, since the depth value is estimated using sparse depth information obtained from the captured real-world scene, and since the new depth value is located on the grid point and used to render the virtual object, the virtual object and the real-world scene can have the same or similar parallax and occlusion. This allows one or more virtual objects and the captured real-world scene to be fused through viewpoint matching to create a final virtual view that is more realistic and beautiful for the user.

[0135] although Fig.10 An example of parallax correction and occlusion overlap 1000 for a virtual object is shown, but may be Fig.10Various changes may be made. For example, the real-world objects captured in various scenes and the virtual objects or other digital content inserted into the various scenes may vary greatly based on the specific application and user environment. In addition, virtual objects or other digital content may be inserted into the various scenes in any other suitable manner.

[0136] Note that the above techniques support the use of viewpoint matching between a perspective camera and a virtual rendering camera using sparse depth information. The above techniques are more efficient than methods that require depth data to be generated for all pixels in the captured image, and are more efficient than methods that perform approximate parallax correction and occlusion overlap. The above techniques can operate using sparse depth information while still achieving efficient parallax correction and occlusion overlap. In addition, the above techniques can be conveniently implemented in a rendering pipeline such as a GPU. This can also avoid transmitting image data back and forth between the CPU and the GPU, which, as described above, can reduce latency and memory consumption. In addition, some of the transformations in the above techniques can be combined with rendering transformations, which means that these transformation operations take up almost no resources in terms of processing resources in the rendering pipeline. In addition, some implementations of the above techniques can reduce or avoid the use of machine learning models, which may be difficult to implement in the rendering pipeline in some cases. Some implementations of the above techniques can be seamlessly combined with the rendering pipeline and implemented with OpenGL shaders.

[0137] As a specific example of the benefits that may be obtained using these techniques, assume that each input image 302 has a resolution of 3,000 pixels by 3,000 pixels. A method that requires dense depth information may require a depth value for each pixel of each input image 302, which means that each input image 302 will have 3,000×3,000 or 9,000,000 depth values. In contrast, assume that each grid 206 is defined as a set of 300 grid points by 300 grid points. In these embodiments, grid processing may involve using 300×300 or 90,000 depth values. Similarly, while a method that requires dense depth information may require depth reprojection for each pixel of each input image 302, each grid 206 is also associated with a depth reprojection at each of its grid points to produce a transformed grid 216. Therefore, the reduction in the number of depth values ​​(achieved via the use of grids 206) can significantly reduce the required depth reprojections and other calculations. Although the above techniques may involve additional operations that are not necessarily performed in other methods, the above techniques can still significantly reduce the total number of operations to be performed, which can reduce the amount of processing and memory resources required and speed up the operation of the electronic device 101.

[0138] Fig.11An example method 1100 for mesh transformation with efficient depth reconstruction and filtering in a through-type AR system according to the present disclosure is shown. For ease of illustration, the method 1100 is described as being composed of Figure 1 The electronic device 101 in the network configuration 100 is executed, wherein the electronic device 101 implements Figure 3 However, the method 1100 may be performed using any other suitable devices and architectures and executed in any other suitable system.

[0139] like Fig.11 As shown, at step 1102, an image captured using one or more imaging sensors is obtained. This may, for example, include the processor 120 of the electronic device 101 obtaining an input image 302 captured using one or more perspective cameras 204, 204a-204b or other imaging sensors 180 of the electronic device 101. At step 1104, position data and depth data associated with the image are obtained. This may, for example, include the processor 120 of the electronic device 101 obtaining position information 304 from an inertial measurement unit, accelerometer, gyroscope or other position sensor 180 of the electronic device 101. This may also include the processor 120 of the electronic device 101 obtaining depth information 306 from one or more depth sensors 180 of the electronic device 101.

[0140] At step 1106, a 3D point cloud representing the environment around the imaging sensor is generated. This may, for example, include: the processor 120 of the electronic device 101 processes the input image 302 and the position information 304 to generate one or more gestures 310 of the one or more perspective cameras 204, 204a-204b or other imaging sensors 180 of the electronic device 101. This may also include: the processor 120 of the electronic device 101 processes the input image 302 and the position information 304 to generate 3D sparse points 312 defined using the input image 302. This may also include: the processor 120 of the electronic device 101 processes the gestures 310 and the 3D sparse points 312 to generate a 3D point cloud representing the environment around the one or more perspective cameras 204, 204a-204b or other imaging sensors 180 of the electronic device 101.

[0141] At step 1108, one or more grids of one or more input images are generated. This may include, for example, the processor 120 of the electronic device 101 processing the input image 302 to identify one or more objects in the input image 302 and generating a grid 206 associated with the input image 302 or an object in the input image 302. At step 1110, one or more depths of one or more grid points in at least one grid are determined. This may include, for example, the processor 120 of the electronic device 101 determining whether each grid point in each grid 206 has a known or measured depth. This may also include: the processor 120 of the electronic device 101 creates at least one depth filter and applies the at least one depth filter to a neighborhood of depth values ​​around each grid point in each grid 206 that does not have a known or measured depth (and may even be applied to those grid points where there is a known or measured depth in order to perform filtering, noise removal, verification, etc.).

[0142] At step 1112, each of the one or more grids is transformed from the viewpoint of the perspective camera to the viewpoint of the virtual rendering camera. This may include, for example, the processor 120 of the electronic device 101 deforming or otherwise transforming the grid 206 into a corresponding transformed grid 216 based on the knowledge of the relationship between the perspective cameras 204, 204a-204b and the virtual rendering camera 212 and the knowledge of the depth in the scene. At step 1114, the left virtual view and the right virtual view are rendered for presentation on the AR device. This may include, for example, the processor 120 of the electronic device 101 constructing the left virtual view and the right virtual view including the digital content to be provided to the user. In some embodiments, this may include the processor 120 of the electronic device 101 using the one or more transformed grids 216 to deform the one or more input images 302 and perform parallax correction and occlusion overlap on the digital content (such as one or more virtual objects) inserted into the input images 302. At step 1116, the left virtual view and the right virtual view are displayed on the AR device. This may, for example, include the processor 120 of the electronic device 101 initiating display of the left virtual view and the right virtual view on the display panel 208 of the electronic device 101 .

[0143] although Fig.11 An example of a method 1100 for mesh transformation with efficient depth reconstruction and filtering in a see-through AR system is shown, but may be used for Fig.11 For example, although shown as a series of steps, Fig.11 The steps in may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0144] Although the present disclosure has been described by way of example embodiments, various changes and modifications may occur to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: Obtaining an image of an environment captured by an imaging sensor associated with a see-through augmented reality (AR) device, position data associated with the image, and depth data associated with the image; generating a point cloud representing the environment based on the image, the position data, and the depth data; generating a grid of a specified image in the image, the grid comprising grid points where grid lines intersect; determining one or more depths of one or more of the grid points of the grid; transforming the grid from a viewpoint of a designated one of the imaging sensors that captured the designated image to a viewpoint of a user of a see-through AR device based on one or more depths of the one or more grid points; as well as A virtual view of the specified image is rendered based on the transformed mesh for presentation by the penetrated AR device.

2. The method according to claim 1, wherein: Determining one or more depths of the one or more grid points includes determining one or more depths of the one or more grid points based on at least one of: image feature information, depth feature information, object pose information, image intensity information, and spatial information.

3. The method according to claim 1, wherein: Generating the point cloud includes generating a point cloud based on: (i) first depth data collected by one or more depth sensors associated with the see-through AR device; and (ii) second depth data determined by applying simultaneous localization and mapping (SLAM) to the designated image and the position data.

4. The method according to claim 1, wherein: Determining one or more depths of the one or more grid points includes determining the depth of the specified grid point based on a weighted average of known depths associated with points in the point cloud within a threshold distance of the specified grid point among the one or more grid points.

5. The method according to claim 4, wherein: The weighted average is determined using a weight based on at least one of: image feature information, depth feature information, object pose information, image intensity information, and spatial information.

6. The method according to claim 1, wherein: Rendering the virtual view includes: determining a parallax of a virtual object and a position of the virtual object relative to one or more real objects in the specified image based on the transformed grid; and A virtual view of the designated image is rendered to include the virtual object based on the determined disparity and the determined position.

7. The method according to claim 1, further comprising: generating different grids for different ones of the images; for each of the grids, determining one or more depths of one or more grid points in the grid, and transforming the grid based on the one or more depths; as well as Different virtual views of different ones of the images are rendered based on different ones of the transformed grids for presentation by the see-through AR device.

8. A penetrating augmented reality (AR) device, comprising: an imaging sensor configured to capture an image of an environment; At least one processing device configured to: obtaining the image, position data associated with the image, and depth data associated with the image; generating a point cloud representing the environment based on the image, the position data, and the depth data; generating a grid of a specified image in the image, the grid comprising grid points where grid lines intersect; determining one or more depths of one or more of the grid points of the grid; transforming the grid from a viewpoint of a designated one of the imaging sensors that captured the designated image to a viewpoint of a user of a see-through AR device based on one or more depths of the one or more grid points; as well as rendering a virtual view of the specified image based on the transformed mesh; as well as At least one display is configured to present the virtual view.

9. The penetrating AR device according to claim 8, wherein: The at least one processing device is configured to determine one or more depths of the one or more grid points based on at least one of: image feature information, depth feature information, object pose information, image intensity information, and spatial information.

10. The penetrating AR device according to claim 8, wherein: The at least one processing device is configured to generate a point cloud based on: (i) first depth data collected by one or more depth sensors associated with the see-through AR device; and (ii) second depth data determined by applying simultaneous localization and mapping (SLAM) to the designated image and the position data.

11. The see-through AR device according to claim 8, wherein: The at least one processing device is configured to determine a depth of a specified grid point among the one or more grid points based on a weighted average of known depths associated with points in the point cloud that are within a threshold distance of the specified grid point.

12. The penetrating AR device according to claim 11, wherein: The at least one processing device is configured to determine the weighted average using a weight based on at least one of: image feature information, depth feature information, object pose information, image intensity information, and spatial information.

13. The see-through AR device according to claim 8, wherein: In order to render the virtual view, the at least one processing device is configured to: determining a disparity of a virtual object and a position of the virtual object relative to one or more real objects in the specified image based on the transformed grid; as well as A virtual view of the designated image is rendered to include the virtual object based on the determined disparity and the determined position.

14. The see-through AR device according to claim 8, wherein: The at least one processing device is configured to: generating different grids for different ones of the images; for each of the grids, determining one or more depths of one or more grid points in the grid, and transforming the grid based on the one or more depths; as well as Different virtual views of different ones of the images are rendered based on different ones of the transformed grids for presentation by the see-through AR device.

15. A machine-readable medium comprising instructions that, when executed, cause at least one processor of a see-through augmented reality (AR) device to: obtaining an image of an environment captured by an imaging sensor associated with a see-through AR device, position data associated with the image, and depth data associated with the image; generating a point cloud representing the environment based on the image, the position data, and the depth data; generating a grid of a specified image in the image, the grid comprising grid points where grid lines intersect; determining one or more depths of one or more of the grid points of the grid; transforming the grid from a viewpoint of a designated one of the imaging sensors that captured the designated image to a viewpoint of a user of the see-through AR device based on one or more depths of the one or more grid points; as well as A virtual view of the specified image is rendered based on the transformed mesh for presentation by the penetrated AR device.

Images