Method for improving pass-through view and host for providing mixed reality service

Abstract

This invention provides a method for improving perspective views and a host for providing mixed reality services. The method includes: acquiring current frame information and historical frame information of the host; determining historical tracking information and texture information of a target object based on the historical frame information of the host; predicting current tracking information of the target object based on the historical tracking information of the target object and the current frame information of the host; rendering an object image corresponding to the target object based on the current tracking information and texture information of the target object; and generating and displaying a perspective view based on the rendered object image, the current frame information of the host, and the current tracking information of the target object.

Description

Technical Field

[0001] The present invention relates to a mechanism for providing reality services, and in particular to a method and a host for improving perspective images. Prior Art

[0002] Mixed Reality (MR) is a technology that combines the real world and the virtual world, allowing virtual objects to interact and merge with objects in the real environment, allowing users to see and manipulate the contents of both worlds at the same time.

[0003] Key features of MR include the fusion of the real and virtual worlds, overlaying virtual objects (such as 3D models and holograms) onto the real world, dynamically adjusting to the user's position and perspective. Furthermore, users can interact with virtual objects, which react to the real-world environment and the user's movements. Through a head-mounted display (HMD) or other device, users experience an immersive experience in which virtual and real content are seamlessly integrated.

[0004] When providing MR services, the HMD can present corresponding visual content to the user. This visual content can include a pass-through view and virtual objects superimposed on the pass-through view. The pass-through view can be generated by the HMD based on image frames (e.g., including the real-world scene in front of the HMD) captured by its front-facing camera (e.g., an RGB camera).

[0005] Generally speaking, after acquiring the image frames captured by the front camera, the HMD needs to project them based on the positions of the user's eyes, generating a left-eye image frame (which can be understood as a perspective image corresponding to the left eye) and a right-eye image frame (which can be understood as a perspective image corresponding to the right eye), respectively corresponding to the user's eyes. The left-eye image frame and the right-eye image frame are then displayed to the user's left and right eyes, respectively, via near-eye displays corresponding to the user's eyes.

[0006] However, since the position of the front lens on the HMD is different from the position of the user's eyes, if the texture of an object in front of the user cannot be captured by the front lens, it may cause texture loss in the perspective image projected by the HMD.

[0007] For ease of understanding, the following explanation will use Figure 1 as an example. Please refer to Figure 1, which is a schematic diagram of texture loss. In the scenario shown in Figure 1, assume that the front camera of the HMD is positioned at the position of the user's right eye. Therefore, when the user makes the gesture shown in Figure 1 with their right hand (for example, a knife-hand gesture with their palm facing left) and places their right hand in front of their right eye, the user's left eye and right eye should ideally see left-eye image frame 101 and right-eye image frame 102, respectively. Furthermore, due to the difference in perspective between the user's two eyes, when the user makes the gesture shown in front of their right eye, the user's left eye should ideally be able to see the texture on the palm of their right hand.

[0008] However, due to the position of the front camera, when the user makes the right-hand gesture shown in front of their right eye, the front camera cannot actually capture the palm of their right hand. In this case, the HMD generates, for example, left-eye image frame 111 and right-eye image frame 112 based on the image frames captured by the front camera. As can be seen from left-eye image frame 111, in this scenario, the user's left eye cannot correctly see the texture of the right palm in left-eye image frame 111 (i.e., the texture on the right palm is missing). This impacts the user's visual experience. Summary of the Invention

[0009] In view of this, the present invention provides a method and a host for improving perspective images, which can be used to solve the above-mentioned technical problems.

[0010] An embodiment of the present invention provides a method for improving perspective images, performed by a host providing a mixed reality service. The method includes: obtaining current per-frame information and historical per-frame information from the host; determining historical tracking information and texture information of a target object based on the historical per-frame information from the host; predicting current tracking information of the target object based on the historical tracking information of the target object and the current per-frame information from the host; rendering an object image corresponding to the target object based on the current tracking information and texture information of the target object; and generating the perspective image based on the rendered object image, the current per-frame information from the host, and the current tracking information of the target object, and displaying the perspective image.

[0011] An embodiment of the present invention provides a host for providing a mixed reality service, comprising a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to execute the following operations: obtaining current per-frame information and historical per-frame information from the host; determining historical tracking information and texture information of a target object based on the historical per-frame information from the host; predicting current tracking information of the target object based on the historical tracking information of the target object and the current per-frame information from the host; rendering an object image corresponding to the target object based on the current tracking information and texture information of the target object; and generating a perspective image based on the rendered object image, the current per-frame information from the host, and the current tracking information of the target object, and displaying the perspective image. Simple diagram description

[0012] Figure 1 is a schematic diagram of texture loss. FIG. 2 is a schematic diagram of a host according to an embodiment of the present invention. FIG. 3 is a flow chart of a method for improving a perspective image according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an application scenario according to an embodiment of the present invention. Implementation Method

[0013] Referring to Figure 2, a schematic diagram of a host according to an embodiment of the present invention is shown. In various embodiments, host 200 can be any device capable of performing tracking functions (e.g., inside-out tracking and / or outside-in tracking) on ​​one or more target objects (also referred to as objects to be tracked, trackable objects, etc., such as the hand of the user of host 200) within the tracking range of host 200. In an embodiment of the present invention, host 200 can be equipped with a tracking camera whose image capture range corresponds to the tracking range. When a target object (e.g., a hand) is within the tracking range, the tracking camera on host 200 can capture an image of the target object, and host 200 can track the posture of each target object based on the captured image, but the present invention is not limited to this.

[0014] In various embodiments, host 200 can be any smart device and / or computer device capable of providing visual content from a real-world service, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or extended reality (XR), but the present invention is not limited thereto. In some embodiments, host 200 can be a head-mounted display (HMD) capable of displaying / providing visual content (e.g., AR / VR / MR content) for viewing by a wearer / user. To better understand the concepts of the present invention, the following assumes that host 200 is an MR device (e.g., an MR HMD) that provides MR content for viewing by a user, but the present invention is not limited thereto.

[0015] In embodiments where the visual content is MR content, the MR content may include a see-through image. In one embodiment, the MR content may also include at least one rendered virtual object overlaid on the see-through image. In this case, the see-through image serves as the underlying image for the visual content, but this is not a limitation.

[0016] In one embodiment, the perspective image can be rendered by, for example, the processor 204 of the host 200 based on image frames (e.g., RGB image frames) captured by, for example, a front-facing camera (e.g., an RGB camera) of the host 200. In this case, a user wearing the host 200 (e.g., an HMD) can see the real-world scene in front of the user through the perspective image in the visual content provided by the host 200.

[0017] In one embodiment, the processor 204 may render one or more virtual objects based on the MR application currently running on the host 200, and the processor 204 may overlay the rendered virtual objects on the rendered perspective image to form / generate visual content (e.g., MR content).

[0018] In one embodiment, the host 200 may be configured with a built-in display (e.g., a near-eye display corresponding to the user's eyes) to display visual content for the user to view. Additionally or alternatively, the host 200 may be connected to one or more external displays, and the host 200 may transmit the visual content to the external displays for display on the external displays, but the present invention is not limited thereto.

[0019] In FIG2 , host 200 includes storage circuit 202 and processor 204. Storage circuit 202 is a combination of one or more static or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, which stores multiple modules executable by processor 204.

[0020] The processor 204 may be coupled to the storage circuit 202. The processor 204 may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), any other type of integrated circuit (IC), a state machine, a graphics processing unit (GPU), etc.

[0021] In an embodiment of the present invention, the processor 204 may access the modules and / or program codes stored in the storage circuit 202 to implement the method for improving the perspective image provided in the present invention, which will be further discussed below.

[0022] Referring to Figure 3, a flowchart illustrating a method for improving perspective images according to one embodiment of the present invention is shown. This method can be executed by host 200 in Figure 2. The details of each step in Figure 3 will be described in conjunction with the components shown in Figure 2. Furthermore, to further enhance understanding of the concepts of the present invention, the following description will be supplemented with Figure 4, which illustrates an application scenario according to one embodiment of the present invention.

[0023] In step S310 , the processor 204 obtains the current frame information 420 and the historical frame information 410 of the host 200 .

[0024] In an embodiment of the present invention, the current frame information 420 may be used, for example, to allow the host 200 to render MR visual content corresponding to the state / context at the current time point, but is not limited thereto.

[0025] In FIG4 , current frame information 420 includes, for example, a current image frame 421 and a current host posture 422. In one embodiment, current image frame 421 is, for example, an image frame captured by the front camera of host 200 of a real-world scene currently located in front of host 200, but is not limited thereto.

[0026] Additionally, current host posture 422 is data obtained by, for example, the host 200 tracking its current posture using an inside-out tracking mechanism and / or an outside-in tracking mechanism. In some embodiments, current host posture 422 may be presented using, for example, six degrees of freedom, but is not limited thereto.

[0027] In some embodiments, the current frame information 420 may also include other information, such as internal parameters and external parameters of the tracking camera. In addition, the current frame information 420 may also include internal parameters and external parameters of the inertial measurement unit (IMU) of the host 200.

[0028] In an embodiment of the present invention, the historical frame information 410 is, for example, information that the host 200 has used to render MR visual content corresponding to a state / context at a past historical time point, but is not limited thereto.

[0029] In FIG4 , historical frame information 410 includes, for example, historical image frames 411 and historical host postures 412. In one embodiment, historical image frames 411 are, for example, image frames previously captured by the front camera of host 200 of a real-world scene currently in front of host 200, but the present invention is not limited thereto.

[0030] In addition, the historical host posture 412 is, for example, data obtained by the host 200 previously tracking the posture of the host 200 itself. In some embodiments, the historical host posture 412 can be presented in the form of six degrees of freedom, for example, but is not limited thereto.

[0031] In one embodiment, assuming that the current per-frame information 420 is per-frame information obtained by the host 200 at the i-th time point (i is a time index value), the historical per-frame information 410 can be understood as per-frame information previously obtained by the host 200 at other time points before the i-th time point.

[0032] For example, current per-frame information 420 may include an image frame captured by the front camera of the host 200 at time point i (referred to as current image frame 421), and the host posture tracked by the host 200 at time point i (referred to as current host posture 422). In this case, historical per-frame information 410 may include image frames captured by the front camera of the host 200 from time point i-1 to time point iK (K may be a positive integer determined by the designer) (referred to as historical image frames 411), and the host postures tracked by the host 200 from time point i-1 to time point iK (referred to as historical host posture 412).

[0033] In step S320 , the processor 204 determines the historical tracking information 431 and texture information 432 of the target object (hereinafter referred to as O) based on the historical frame information 410 of the host 200 .

[0034] For ease of understanding, the following assumes that the target object O is a user's hand (e.g., the user's right hand in the scenario of FIG1 ). In other embodiments, the target object O can be adjusted to various objects trackable by the host 200 based on the designer's needs, such as handheld controllers, various trackers, and trackable furniture and furnishings, but is not limited thereto.

[0035] In FIG. 4 , the processor 204 may determine the historical tracking information 431 and texture information 432 of the target object O by, for example, executing a reconstruction algorithm A01 based on a historical image frame 411 and a corresponding historical host pose 412 .

[0036] In an embodiment of the present invention, processor 204, which applies reconstruction algorithm A01, generates a corresponding point cloud using back-projection (2D to 3D un-projection) when acquiring each image frame and its corresponding host pose. After accumulating point clouds from multiple frames (historical image frames 411), processor 204 can then combine these with various previously collected object kinematic models to optimize and predict point cloud correspondences, thereby improving the kinematic model results.

[0037] By combining the correspondence between multiple point clouds and RGB images, a textured mesh model of the target object O (and / or other trackable objects) can be simplified and optimized. This mesh model can serve as the texture information 432 in FIG. 4 . The processor 204 can then use the optimized dynamic model results to analyze and determine precise tracking information for the target object O (and / or other trackable objects), thereby determining historical tracking information 431 for the target object O.

[0038] In different embodiments, the reconstruction algorithm A01 may be implemented using, for example, multi-view stereo, truncated signed distance function fusion, neural radiance fields, and other technical means, but is not limited thereto.

[0039] In step S330 , the processor 204 predicts current tracking information 440 of the target object O (eg, the current posture of the target object O) based on the historical tracking information 431 of the target object O and the current frame information 420 of the host 200 .

[0040] In FIG. 4 , the processor 204 may, for example, execute a tracking and prediction algorithm A02 based on the historical tracking information 431 and the current frame information 420 to predict the current tracking information 440 of the target object O.

[0041] In an embodiment of the present invention, the processor 204 applying the tracking and prediction algorithm A02 may utilize certain motion prediction techniques to predict the current tracking information 440 of the target object O based on the historical tracking information 431 and the current frame information 420 .

[0042] In different embodiments, the tracking and prediction algorithm A02 may be implemented using, for example, a Kalman filter, a particle filter, or other technical means, but is not limited thereto.

[0043] In step S340 , the processor 204 renders an object image 450 corresponding to the target object O based on the current tracking information 440 and the texture information 432 of the target object O.

[0044] In FIG. 4 , the processor 204 may, for example, execute a rendering algorithm A03 based on the current tracking information 440 and the texture information 432 of the target object O to render an object image 450 corresponding to the target object O.

[0045] In one embodiment, the processor 204 applying the rendering algorithm A03 may use, for example, a 3D to 2D projection technique to render an object image 450 corresponding to the target object O based on the current tracking information 440 and texture information 432 of the target object O.

[0046] In different embodiments, the rendering algorithm A03 may be implemented using, for example, rasterization, ray tracing, and other technical means, but is not limited thereto.

[0047] Since the processor 204 can render the object image 450 (eg, the image of a hand) based on the texture information 432, the object image 450 can still present the corresponding texture, thereby avoiding the texture loss problem mentioned above.

[0048] In one embodiment, the texture corresponding to at least a portion of the rendered object image 450 does not exist in the current image frame 421. Specifically, for the front-facing camera of the host 200, at least a portion of the target object O (e.g., the right palm in the scenario of FIG1 ) may not be directly captured by the front-facing camera in the current image frame 421. However, because the processor 204 renders the object image 450 based on the texture information 432, even if at least a portion of the target object O is not captured in the current image frame 421, the rendered object image 450 can still correctly present the corresponding texture.

[0049] In step S350 , the processor 204 generates a perspective image 460 based on the rendered object image 450 , the current frame information 420 of the host 200 , and the current tracking information 440 of the target object O, and displays the perspective image 460 .

[0050] In FIG. 4 , the processor 204 may generate a perspective image 460 based on, for example, a superposition algorithm A04.

[0051] In one embodiment, the processor 204, which applies the folding algorithm A04, may determine a trackable portion and an untrackable portion within the current image frame 421 based on the current image frame 421 and the current tracking information 440 of the target object O. In various embodiments, the trackable portion may include various trackable objects, and the untrackable portion may include various untrackable objects, such as background objects, but is not limited thereto.

[0052] In the embodiment of FIG. 4 , the trackable portion corresponds to the target object O, for example, but is not limited thereto.

[0053] Thereafter, the processor 204 may execute an image-based rendering algorithm based on the untrackable portion to generate a reference frame. Details of the image-based rendering algorithm may be found in literature related to triangular mesh-based image warping and are not further described herein.

[0054] After generating the reference frame, the processor 204 may overlay the rendered object image 450 onto the reference frame based on the current tracking information 440 of the target object O to generate a perspective frame 460 .

[0055] As previously mentioned, since there is no texture loss problem on the object image 450, after the processor 204 superimposes the object image 450 on the reference frame to generate the perspective frame 460, the image area corresponding to the object image 450 in the perspective frame 460 can avoid the texture loss problem.

[0056] Taking FIG. 1 as an example, after applying the method proposed in an embodiment of the present invention, the user's left eye and right eye can respectively view the left-eye image frame 101 and the right-eye image frame 102, thereby avoiding the user having a poor visual experience due to viewing image frames with texture loss.

[0057] In summary, the method proposed in the embodiments of the present invention allows the host to determine the historical tracking and texture information of a target object based on each frame of historical information. The host then renders an object image based on the target object's texture information and the predicted current tracking information. This rendered object image is then overlaid onto the current image frame captured by the front-facing camera to generate a perspective image. Because the object image is rendered based on this texture information, texture loss in the image area corresponding to the target object in the perspective image is avoided, thereby improving the user's visual experience.

[0058] Although the present invention has been disclosed above with reference to the embodiments, they are not intended to limit the present invention. Anyone with ordinary skill in the art may make slight changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the appended patent applications.

[0059] 101, 111: left eye image frame 102, 112: right eye image frame 200: Host 202: Storage circuit 204: Processor S310, S320, S330, S340, S350: Steps 410: Historical frame information 411: Historical Image Frame 412: Historical host status 420: Current frame information 421: Current image frame 422: Current host status 431: Historical Tracking Information 432: Texture Information 440: Current tracking information 450: Object Image 460: Perspective Picture A01: Reconstruction Algorithm A02: Tracking and Prediction Algorithms A03: Rendering Algorithm A04: Overlap Algorithm

Claims

1. A method for improving perspective, performed by a host providing mixed reality services, the method comprising: The processor of the host obtains the current frame information and historical frame information of the host, wherein the current frame information of the host includes the current image frame captured by the front camera of the host and the current host posture corresponding to the current image frame; the processor of the host determines the historical tracking information and texture information of the target object based on the historical frame information of the host. The processor of the host predicts the current tracking information of the target object based on the historical tracking information of the target object and the current frame information of the host. The processor of the host computer renders an object image corresponding to the target object based on the current tracking information and texture information of the target object; The processor of the host computer generates the perspective view based on the rendered object image, the current frame information of the host computer, and the current tracking information of the target object, and displays the perspective view.

2. The method as described in claim 1, wherein the historical frame information of the host includes multiple historical image frames captured by the front camera of the host and multiple historical host poses corresponding to the multiple historical image frames, and the processor of the host determines the historical tracking information and texture information of the target object based on the historical frame information of the host, including: The processor of the host computer determines the historical tracking information and texture information of the target object by performing a reconstruction algorithm based on the plurality of historical image frames and the corresponding plurality of historical host poses.

3. The method as described in claim 1, wherein generating the perspective view by the processor of the host based on the rendered object image, the current frame information of the host, and the current tracking information of the target object includes: The processor of the host determines the trackable and untrackable portions within the current image frame based on the current image frame and the current tracking information of the target object, wherein the trackable portion corresponds to the target object; The host's processor performs a fixed-depth rendering algorithm based on the untraceable portion to generate a reference image; The processor of the host computer overlays the rendered object image onto the reference screen based on the current tracking information of the target object to generate the perspective view.

4. The method as described in claim 1, wherein at least a portion of the rendered object image corresponds to a texture that does not exist in the current image frame.

5. A host for providing mixed reality services, comprising: A storage circuit that stores a piece of code; And a processor coupled to the storage circuit and accessing the code to execute: obtaining current frame information and historical frame information of the host, wherein the current frame information of the host includes a current image frame captured by the host's front camera and the current host pose corresponding to the current image frame; determining historical tracking information and texture information of a target object based on the historical frame information of the host; and predicting current tracking information of the target object based on the historical tracking information of the target object and the current frame information of the host. Render an object image corresponding to the target object based on the current tracking information and texture information of the target object; and generate a perspective view based on the rendered object image, the current frame information of the host and the current tracking information of the target object, and display the perspective view.

6. The host as claimed in claim 5, wherein the historical frame information of the host includes a plurality of historical image frames captured by the front camera of the host and a plurality of historical host poses corresponding to the plurality of historical image frames, and the processor is configured to perform: determining the historical tracking information and the texture information of the target object by performing a reconstruction algorithm based on the plurality of historical image frames and the corresponding plurality of historical host poses.

7. The host as claimed in claim 5, wherein the processor is configured to perform: determining, based on the current image frame and the current tracking information of the target object, a trackable portion and an untrackable portion within the current image frame, wherein the trackable portion corresponds to the target object; A fixed-depth rendering algorithm is executed based on the untraceable portion to generate a reference image; And based on the current tracking information of the target object, the rendered image of the object is overlaid onto the reference image to generate the perspective view.

8. The host as described in claim 5, wherein at least a portion of the rendered object image corresponds to a texture that does not exist in the current image frame.

Images