XR distribution method and XR projection system

Abstract

An XR delivery method for delivering XR images from an Extended Reality (XR) application instance to an XR device. [Solution] An XR application instance is incorporated into a computer device provided separately from the XR device. The XR delivery method includes the steps of: drawing an XR image for display on the XR device with the XR application instance relative to a predetermined reference position, the predetermined reference position having a predetermined relationship with the instantaneous position of the XR device; delivering the drawn XR image to the XR device; and reprojecting the drawn XR image to the position of the first eye and / or the position of the second eye by the XR device, so that a first XR image for the first eye and a second XR image for the second eye are obtained based on the drawn XR image.

Description

Technical Field

[0001] The present invention generally relates to an XR distribution method for distributing XR images from an XR application instance to an XR device. The present invention further relates to an XR projection system.

Background Art

[0002] Certain types of extended reality applications require two images, namely, a first image for the user's left eye and a second image for the user's right eye, in order to enable a realistic 3D experience for the user.

[0003] Since rendering these images is computationally expensive, these images may be rendered by an external computer device and then distributed to the user's XR device.

[0004] However, the necessary distribution bandwidth increases because two images are transmitted per frame.

Summary of the Invention

[0005] An object of the present invention is to provide an XR distribution method and an XR projection system that do not require much distribution bandwidth.

[0006] The present invention solves the problem with an XR distribution method for distributing XR images from an Extended Reality (XR) application instance to an XR device. The XR application instance is incorporated into a computer device provided separately from the XR device. The XR distribution method includes the steps of: the XR application instance drawing an XR image for display on the XR device relative to a predetermined reference position, the predetermined reference position having a predetermined relationship with the instantaneous position of the XR device; the XR application instance distributing the drawn XR image to the XR device; and the XR device reprojecting the drawn XR image to the position of the first eye and / or the XR device reprojecting the drawn XR image to the position of the second eye, such that a first XR image for the first eye and a second XR image for the second eye are obtained based on the drawn XR image.

[0007] In the foregoing and below, the term “location” is understood to describe a place and / or direction.

[0008] Therefore, the term "position of the first eye" refers to the location and / or direction of the user's first eye.

[0009] The term "XR device" is understood to describe an electronic device configured to display Extended Reality (XR) images, i.e., Augmented Reality (AR), Mixed Reality (MR), and / or Virtual Reality (VR) images.

[0010] For example, the XR device may be a head-mounted display, or, for example, a pair of glasses-type electronic wearables. However, it is understood that the XR device may also be any other XR-capable electronic device, such as a smartphone or tablet.

[0011] Furthermore, the term "XR image" is understood to describe at least one virtual image.

[0012] In the case of augmented reality (AR) or mixed reality, the XR image corresponds to at least one virtual image that is superimposed on reality.

[0013] For example, an XR device may be a head-mounted display with a semi-transparent screen, where the virtual image is displayed on the semi-transparent screen so that the user can directly see the environment along with the overlaid virtual image through the semi-transparent screen.

[0014] As another example, the XR device may be an optically opaque head-mounted display. In this case, the head-mounted display may include at least one internal camera, in particular several internal cameras configured to capture images of the environment of the head-mounted display. The reality images captured by the internal cameras are superimposed on a virtual image, and the result of the superimposition of the reality image and the extended reality image is displayed on the screen of the head-mounted display.

[0015] As another example, the XR device may be a smartphone or a tablet, and an image captured by the XR device's camera is superimposed onto at least one virtual image, and the resulting image is displayed on the screen.

[0016] In the case of virtual reality, an XR image corresponds to a virtual image displayed on the screen of an XR device.

[0017] For example, the XR device may be an optically opaque head-mounted display. The VR image may be displayed on the screen of the head-mounted display.

[0018] The XR distribution method according to the present invention is based on the idea of ​​using reprojection technology to save distribution bandwidth.

[0019] Instead of drawing an XR image to both eyes in each frame and delivering both images to the XR device, only a single XR image is drawn and sent to the XR device in each frame.

[0020] Based on a single XR image, a first XR image for the first eye and / or a second XR image for the second eye are obtained by reprojecting the delivered XR image.

[0021] Generally, a predetermined reference position is any position that has a certain relationship with the instantaneous position of the XR device.

[0022] As will be explained in more detail below, the rendered XR image may correspond to the position of the first eye or the position of the second eye. In this case, the rendered XR image may be reprojected to either the position of the second eye or the position of the first eye, respectively.

[0023] Alternatively, the rendered XR image may be located at a position different from both the first eye position and the second eye position. In this case, the rendered XR image may be reprojected onto both the first eye position and the second eye position.

[0024] For example, the predetermined reference position may be the average of the positions of the first eye and the second eye.

[0025] In the XR distribution method according to the present invention, the required distribution bandwidth can be reduced significantly, or up to 50%.

[0026] Furthermore, the rendered XR image received by the XR device may be further reprojected based on the updated position of the XR device, and this position is different from the instantaneous position of the XR device where the XR image is rendered. In the above, any suitable reprojection technique may be used, for example, reprojection-based machine learning and / or algorithm-based reprojection techniques.

[0027] In other words, the first and second XR images may be reprojected based on the updated position of the XR device, so as to take into account the position of the XR device and / or the user.

[0028] This enables providing a stable world anchor hologram even when there is a large difference between the instantaneous position of the XR device and the updated position of the XR device when at least one XR image is being displayed. For example, such a large difference in position can occur when the user's head moves quickly.

[0029] Furthermore, the reprojection method according to the present invention provides a stable world anchor hologram even when there is a large delay, for example, a delay of 100 ms or more. This delay can be a further cause of a large difference between the instantaneous position at the time of rendering and the position of the XR device when at least one XR image is being displayed.

[0030] Note that the steps of the above method relate to a single frame. Of course, these steps may be repeatedly executed so that a continuous delivery of XR images to the XR device is obtained.

[0031] According to an aspect of the present invention, the predetermined reference position is the instantaneous position of the first eye or the instantaneous position of the second eye. Therefore, the rendered XR image may correspond to the first XR image or the second XR image. The second XR image or the first XR image is obtained by reprojection of the rendered XR image from the position of the first eye to the position of the second eye or from the position of the second eye to the position of the first eye, respectively.

[0032] Therefore, for one of the position of the first eye and the position of the second eye, the rendered XR image may be reprojected only based on the updated position of the first eye or the updated position of the second eye, taking into account the movement of the user and / or the XR device. This leads to a particularly high quality of the reprojected XR image.

[0033] In an embodiment of the present invention, the predetermined reference position alternates between the instantaneous position of the first eye and the instantaneous position of the second eye. In fact, the predetermined reference position may correspond to the instantaneous position of the first eye for a certain number of frames. Thereafter, the reference position is adjusted to the instantaneous position of the second eye for a certain number of frames, etc.

[0034] According to another aspect of the present invention, a predetermined reference position alternates between the instantaneous position of the first eye and the instantaneous position of the second eye for each consecutive drawn XR image. In other words, the predetermined reference position may switch between the instantaneous position of the first eye and the instantaneous position of the second eye for each consecutive frame.

[0035] Therefore, the XR image may be rendered at half the frame rate of the XR device for each instantaneous first eye position and instantaneous second eye position, and a predetermined reference position for rendering the XR image switches between the instantaneous first eye position and the instantaneous second eye position.

[0036] In fact, when the current drawn XR image corresponds to the position of the second eye, the first XR image for the first eye may be obtained based on the previous drawn XR image, and when the current drawn XR image corresponds to the position of the first eye, the second XR image for the second eye may be obtained based on the previous drawn XR image.

[0037] Therefore, in order to obtain the first or second XR image (i.e., up to the reprojection that takes into account the updated position of the first or second eye), the drawn XR image does not necessarily need to be reprojected in the current frame, and in each frame, the drawn XR image corresponds to either the position of the first eye or the position of the second eye.

[0038] In fact, a second XR image or a first XR image, i.e., an XR image that does not correspond to the drawn XR image received in the current frame, may be obtained by reprojecting the drawn XR image received by the XR device in the previous frame, which corresponds to the position of the second eye or the position of the first eye.

[0039] Alternatively, a second XR image or a first XR image, i.e., an XR image that does not correspond to the drawn XR image received in the current frame, may be obtained by reprojecting the drawn XR image received in the current frame to the position of the second eye or the position of the first eye, respectively. Optionally, the reprojected XR image may be modified based on the drawn XR image received in the previous frame.

[0040] Generally, this results in improved image quality for both the first and / or second XR images.

[0041] Another aspect of the present invention provides that a predetermined reference position is equal to the position of a first eye or a second eye, and the XR delivery method further includes the steps of: drawing an XR image for display at the position of the second eye or the first eye by an XR application instance, thereby obtaining a comparison XR image; reprojecting the drawn XR image to the position of the second eye or the first eye by an XR application instance, thereby obtaining a reprojected XR image; comparing the reprojected XR image with the comparison XR image by an XR application instance, thereby obtaining difference data; and transmitting the difference data to an XR device by an XR application instance.

[0042] In other words, the reprojection performed by the XR device may also be performed by an XR application instance embedded in an external computer device, thereby obtaining a comparison XR image. In fact, the XR application instance may use the same reprojection technique as the XR device.

[0043] The reprojected image is compared to the actual drawn image, i.e., the comparison XR image, at its position, so that the difference data indicates the error introduced to the reprojected XR image by the reprojection. When the XR application instance and the XR device may use the same reprojection technique, the difference data similarly describes the expected error caused by the reprojection performed by the XR device.

[0044] The rendered XR image delivered to the XR device may alternate between the position of the first eye and the position of the second eye, as described above. The reprojection performed by the XR application instance may switch accordingly from the position of the first eye to the position of the second eye, and from the position of the second eye to the position of the first eye.

[0045] Generally, difference data requires significantly less bandwidth compared to complete XR images. Therefore, even when sending difference data to an XR device, the required bandwidth is significantly reduced compared to sending two drawn XR images to the XR device.

[0046] Furthermore, the image quality of XR images acquired by XR devices based on reprojection can be significantly improved based on difference data.

[0047] A second XR image for the second eye, acquired by the XR device, may be modified by the XR device based on difference data, or a first XR image for the first eye, acquired by the XR device, may be modified by the XR device based on difference data. In other words, an XR image acquired by the XR device based on reprojection may be modified based on difference data.

[0048] Therefore, the image quality of XR images acquired through reprojection is improved based on the difference data.

[0049] According to an aspect of the present invention, the difference data is encoded by the XR application instance before being transmitted to the XR device. This can further reduce the distribution bandwidth required to transmit the difference data.

[0050] The XR device may decode the encoded difference data and thereby reconstruct the difference data.

[0051] In embodiments of the present invention, a predetermined reference position is different from the position of the first eye and different from the position of the second eye, and the XR distribution method further includes the steps of: drawing an XR image to be displayed at the position of the first eye by an XR application instance and thereby obtaining a first comparison XR image; drawing an XR image to be displayed at the position of the second eye by an XR application instance and thereby obtaining a second comparison XR image; reprojecting the drawn XR image at the position of the first eye by an XR application instance and thereby obtaining a first reprojected XR image; reprojecting the drawn XR image at the position of the second eye by an XR application instance and thereby obtaining a second reprojected XR image; comparing the first reprojected XR image with the first comparison XR image by an XR application instance and thereby obtaining first difference data; comparing the second reprojected XR image with the second comparison XR image by an XR application instance and thereby obtaining second difference data; and transmitting the difference data to an XR device by an XR application instance.

[0052] In the above, it is understood that the term "difference" data describes both the first difference data and the second difference data.

[0053] In other words, the reprojection performed by the XR device may also be performed by an XR application instance embedded in an external computer device. In fact, the XR application instance may use the same reprojection technique as the XR device.

[0054] The reprojected image is compared to the images actually drawn at each location, i.e., the first and second comparison images, so that the difference data indicates the error that the reprojection brings to each XR image.

[0055] Difference data requires significantly less bandwidth compared to a complete XR image. Therefore, even when sending difference data to an XR device, the required bandwidth is significantly reduced compared to sending two drawn XR images to the XR device.

[0056] Furthermore, the image quality of XR images acquired by XR devices based on reprojection can be significantly improved based on difference data.

[0057] In a further embodiment of the present invention, a first XR image for a first eye acquired by an XR device is modified by the XR device based on first difference data, and / or a second XR image for a second eye acquired by an XR device is modified by the XR device based on second difference data. In other words, an XR image acquired by an XR device based on reprojection may be modified based on difference data.

[0058] Therefore, the image quality of XR images acquired through reprojection is improved based on the difference data.

[0059] The first difference data may be encoded by the XR application instance before transmission to the XR device. Alternatively, or in addition, the second difference data may be encoded by the XR application instance before transmission to the XR device. This can further reduce the distribution bandwidth required to transmit the difference data.

[0060] The XR device may decode the encoded difference data to reconstruct the difference data, more precisely, the first difference data and the second difference data.

[0061] In embodiments of the present invention, the XR application instance and the XR device use the same reprojection algorithm and / or the same machine learning model trained to perform the reprojection. This ensures that the difference data accurately represents the errors of the reprojection performed by the XR device. Thus, it is ensured that these errors can be reliably corrected based on the difference data.

[0062] According to another aspect of the present invention, the drawn XR image is reprojected based on the updated position of the XR device. More precisely, the updated position of the XR device may be taken into consideration when acquiring the first and second XR images.

[0063] The updated position may differ from the instantaneous position of the XR device where the XR image is rendered due to the user's movement and / or the movement of the XR device. This difference is taken into account by the reprojection performed by the XR device.

[0064] According to the present invention, the problem is further solved by an XR projection system. The XR projection system comprises an XR device and an XR application instance, the XR application instance being incorporated into a computer device provided separately from the XR device. The XR projection system is configured to perform an XR delivery method according to any of the above-described forms.

[0065] For the advantages and further characteristics of the XR projection system, refer to the explanation given above regarding the XR delivery method, which also applies to the XR projection system and vice versa.

[0066] According to aspects of the present invention, the computer device is at least one server or includes at least one server. In particular, the computer device may be a server cloud having multiple servers coordinating to provide an XR distribution service, or may include a server cloud.

[0067] However, it is understood that computer devices may be provided as, or include, any other suitable type of computer device, such as a personal computer, laptop, notebook, Mac, tablet, smartphone, or other type of smart device. [Brief explanation of the drawing]

[0068] Many of the aforementioned embodiments and the associated advantages protected by the present invention will be more readily understood by referring to the following detailed description in conjunction with the accompanying drawings.

[0069] [Figure 1] Figure 1 schematically shows the XR projection system according to the present invention. [Figure 2] Figure 2 schematically shows an exemplary embodiment of the XR projection system shown in Figure 1. [Figure 3] Figure 3 shows a flowchart of the XR distribution method according to the present invention. [Modes for carrying out the invention]

[0070] The detailed description below, in conjunction with the accompanying drawings, uses the same number to indicate the same element and is intended to describe various embodiments of the disclosed protected subject matter, and is not intended to show embodiments only. Each embodiment described in this disclosure is provided merely as an example or for illustrative purposes and should not be construed as being preferable or advantageous to other embodiments. The exemplary examples provided herein are not intended to be exhaustive or to limit the claimed protected subject matter to the disclosed forms themselves.

[0071] For the purposes of this disclosure, the phrase "at least one of A, B, and C" means, for example, (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C), including all other possible permutations when three or more elements are listed. In other words, the term "at least one of A and B" generally means "A and / or B," i.e., "A" only, "B" only, or "A and B."

[0072] Figure 1 schematically shows a block diagram of an XR projection system 10, which includes an XR device 12 and an external computer device 14 connected to the XR device 12 by a signal transmission method.

[0073] In the foregoing and below, the term “connected by a signaling scheme” is understood to describe a wired or wireless connection configured to transmit signals between each device or component.

[0074] Generally, an XR device 12 is an electronic device configured to display Extended Reality (XR) images, i.e., Augmented Reality (AR) images, Mixed Reality (MR) images, and / or Virtual Reality (VR) images.

[0075] As shown in Figure 2, the XR device 12 may be configured, for example, as a head-mounted display, and in particular as an electronic wearable having glasses.

[0076] However, it is understood that the XR device 12 may be provided as any other XR-capable electronic device, such as a smartphone or tablet.

[0077] Generally, the XR projection system 10 allows users to observe and / or interact with virtual objects, particularly virtual 3D objects.

[0078] If the XR projection system 10 is an AR projection system or an MR projection system, these virtual objects are embedded in the user's real environment.

[0079] Therefore, the XR device 12 has at least one projection surface 16, and at least one XR image to be displayed is projected onto the projection surface 16 so that a virtual object is displayed to the user.

[0080] The projection surface 16 may be the screen of the XR device 12.

[0081] Optionally, the XR projection system 10 may include at least one handheld input device (not shown in Figure 1), and the user may control the XR projection system 10 using at least one handheld input device.

[0082] For example, at least one handheld input device may be provided as a pen-type device or as any other suitable type of input device.

[0083] The XR device 12 further comprises at least one camera 18, at least one position sensor 20, a communication module 22, and a machine learning module.

[0084] In the foregoing and below, the term “module” is understood to describe suitable hardware, suitable software, or a combination of hardware and software configured to have a specific function.

[0085] The hardware may include, among other things, a CPU, GPU, FPGA, ASIC, or other types of electronic circuits.

[0086] At least one camera 18 is configured to capture images of the environment of the XR device 12, particularly the environment in front of the XR device 12.

[0087] In particular, at least one camera 18 may be provided as a stereo camera. Alternatively, or in addition, the XR device 12 may include several cameras with overlapping fields of view. Thus, depth information of images captured by at least one camera 18 can be determined based on images captured by at least one camera 18.

[0088] At least one position sensor 20 is configured to determine the position, i.e., location and / or orientation, of the XR device 12.

[0089] For example, at least one position sensor 20 may be provided as a gyroscope, accelerometer, magnetometer, GNSS device, and / or any other suitable type of position sensor.

[0090] Generally, the external computer device 14 may be provided as any type of electronic computing device configured to have the functions described below.

[0091] For example, the external computer device 14 may be provided as a server having suitable hardware and suitable software to run on that hardware.

[0092] As another example, the external computer device 14 may be provided as a personal computer, laptop, notebook, Mac, tablet, smartphone, or any other type of smart device.

[0093] The external computer device 14 includes at least one XR application instance 26 and a communication module 28.

[0094] Optionally, the external computer device 14 includes an analysis module 30.

[0095] The functions of each module are described in more detail below.

[0096] The communication module 22 of the XR device 12 and the communication module 28 of the external computer device 14 are configured to communicate with each other.

[0097] In fact, image data, location data, and / or control data may be exchanged between the XR device 12 and the external computer device 14 by communication modules 22, 28.

[0098] In the above, any suitable wireless or wired transmission technology may be used by the communication modules 22, 28, for example, WLAN, 4G, 5G, Ethernet, etc.

[0099] It should be understood that the embodiments of the XR projection system 10 shown in Figures 1 and 2 are illustrative.

[0100] The XR projection system 10 is configured to perform an XR distribution method that delivers XR images from at least one XR application instance 26 to the XR device 12, which is described below with reference to Figure 3.

[0101] Instantaneous position data is determined, and here, instantaneous position data is associated with the instantaneous position of the XR device 12 (step S1).

[0102] Generally, instantaneous location data includes information about the instantaneous location of the XR device 12 and / or information about the instantaneous orientation of the XR device 12.

[0103] Preferably, the instantaneous position data includes information about the instantaneous location of the XR device 12 and information about the instantaneous orientation of the XR device 12, so as to uniquely determine the field of view of the user of the XR device 12 or the XR device 12 itself.

[0104] In the exemplary embodiments shown in Figures 1 and 2, instantaneous position data is determined by at least one position sensor 20 of the XR device 12.

[0105] Alternatively, or in addition, instantaneous positional data may be determined by at least one external camera, i.e., at least one camera not incorporated into the XR device 12.

[0106] At least one external camera may be connected to an analysis module 30, which is configured to determine instantaneous positional data based on images of the XR device 12 captured by at least one external camera. The analysis module 30 may be integrated into the XR device 12 or an external computer device 14.

[0107] Optionally, the XR device 12 may include one or more light-emitting elements, in particular one or more LEDs. The analysis module 30 may determine instantaneous positional data based on images of one or more light-emitting elements captured by at least one external camera, in particular through triangulation.

[0108] Furthermore, it is understood that the analysis module 30 may determine instantaneous position data based on images captured by at least one external camera.

[0109] Instantaneous location data is transmitted to at least one XR application instance 26 via communication modules 22 and 28.

[0110] The XR image to be transmitted to the XR device 12 is acquired by at least one XR application instance 26 based on instantaneous position data, and the XR image is drawn relative to a predetermined reference position (step S2).

[0111] The XR image drawn relative to a predetermined reference position is transmitted to the XR device 12, as will be described in more detail below.

[0112] A predetermined reference position has a predetermined relationship with the instantaneous position of the XR device 12.

[0113] For example, a predetermined reference position may correspond to the position of the first eye, i.e., the location and / or direction of the user's first eye.

[0114] As another example, a predetermined reference position may correspond to the position of the second eye, i.e., the location and / or direction of the user's second eye.

[0115] In fact, the predetermined reference position may alternate between the position of the first eye and the position of the second eye for a continuously drawn XR image.

[0116] In particular, the predetermined reference position may alternate between the instantaneous position of the first eye and the instantaneous position of the second eye for each consecutive frame.

[0117] As another example, the predetermined reference position may be different from both the position of the first eye and the position of the second eye. In particular, in this example, the predetermined reference position may be the average of the positions of the first eye and the second eye.

[0118] However, the predetermined reference position may be any position that has a certain relationship with the instantaneous position of the XR device 12.

[0119] Optionally, the XR image to be displayed on the XR device 12 may be drawn at both the position of the first eye and the position of the second eye (step S3).

[0120] Therefore, when a predetermined reference position corresponds to the position of the first eye, and an XR image to be transmitted to the XR device 12 is drawn at the position of the first eye, the XR image may be additionally drawn at the position of the second eye by at least one XR application instance 26, thereby obtaining a second comparison XR image.

[0121] When a predetermined reference position corresponds to the position of the second eye, and an XR image to be transmitted to the XR device 12 is drawn at the position of the second eye, the XR image may be further drawn at the position of the first eye by at least one XR application instance 26, thereby obtaining a first comparison XR image.

[0122] When a predetermined reference position corresponds to a position different from both the position of the first eye and the position of the second eye, the XR image may be additionally drawn on the positions of the first eye and the second eye by at least one XR application instance 26, thereby obtaining a first comparative XR image and a second comparative XR image.

[0123] The XR image drawn relative to a predetermined reference position is reprojected by at least one XR application instance 26 to the position of the first eye and / or the position of the second eye (step S4).

[0124] More precisely, when a predetermined reference position corresponds to the position of the first eye, the XR image to be transmitted to the XR device 12 may be reprojected onto the position of the second eye, thereby obtaining a second reprojected XR image.

[0125] When a predetermined reference position corresponds to the position of the second eye, the XR image to be transmitted to the XR device 12 may be reprojected onto the position of the first eye, thereby obtaining a first reprojected XR image.

[0126] When a predetermined reference position corresponds to a position different from both the position of the first eye and the position of the second eye, the XR image to be transmitted to the XR device 12 may be reprojected onto both the position of the first eye and the position of the second eye, thereby obtaining a first reprojected XR image and a second reprojected XR image.

[0127] As described above, at least one comparison XR image may be compared with the corresponding reprojected XR image by at least one XR application instance to obtain difference data (step S5).

[0128] More precisely, when a predetermined reference position corresponds to the position of the first eye, the difference data is obtained by comparing the second reprojected XR image with a second comparison XR image.

[0129] When a predetermined reference position corresponds to the position of the second eye, difference data is obtained based on a comparison of the first reprojected XR image with the first comparison XR image.

[0130] When a predetermined reference position corresponds to a position different from both the position of the first eye and the position of the second eye, the difference data includes first difference data and second difference data. The first difference data is obtained based on a comparison of the first reprojected XR image with the first comparison XR image. The second difference data is obtained based on a comparison of the second reprojected XR image with the second comparison XR image.

[0131] The XR image drawn relative to a predetermined reference position, and optionally the difference data, are delivered to the XR device 12 via communication modules 28 and 22 (step S6).

[0132] In the above, the XR image may be encoded before being transmitted to the XR device 12.

[0133] Optionally, the difference data may be encoded before being transmitted to the XR device 12.

[0134] Furthermore, the XR image or the XR image data including the XR image transmitted to the XR device 12 may include information about the view matrix and / or projection matrix.

[0135] When the XR image to be displayed includes virtual world anchor objects, particularly virtual 3D objects, the view matrix and / or projection matrix contain useful information for accurately displaying the XR image on the XR device 12.

[0136] The view matrix contains all the information necessary to transform an object from world space to view space, and the view space is associated with the XR device 12, and more particularly with at least one camera 18 of the XR device 12. In view space, at least one camera 18 may remain at the origin, and in particular, the primary field of view direction of at least one camera is along the z-axis.

[0137] In particular, the view matrix may be determined based on the determined instantaneous position of the XR device 12, or at least one camera 18 of the XR device 12.

[0138] Similarly, the view matrix may be determined based on the determined and updated position of the XR device 12, or at least one camera 18 of the XR device 12.

[0139] The projection matrix contains all the information necessary to project an object from view space into projection space, and projection space is associated with the XR device 12, and more specifically, with at least one of its cameras 18. Generally, projecting an object into this projection space allows the XR device 12 to accurately display the virtual (3D) object.

[0140] The received XR image is reprojected by the XR device 12 onto the position of the first eye and / or the position of the second eye, thereby obtaining a first XR image for the first eye and a second XR image for the second eye (step S7).

[0141] More precisely, the received XR image may be reprojected by the machine learning module 24, which may include a machine learning model pre-trained to perform the reprojection.

[0142] However, it is understood that reprojection may be performed based on a reprojection algorithm, either instead or in addition.

[0143] When the received XR image corresponds to the position of the first eye, the XR image received by the XR device 12 may be the first XR image. The received XR image is reprojected by the XR device 12 to the position of the second eye, thereby obtaining the second XR image.

[0144] When the received XR image corresponds to the position of the second eye, the XR image received by the XR device 12 may be the second XR image. The received XR image is reprojected by the XR device 12 to the position of the first eye, thereby obtaining the first XR image.

[0145] As already mentioned above, the predetermined reference position may alternate between the position of the first eye and the position of the second eye. In this case, when the currently received XR image corresponds to the position of the second eye, the first XR image for the first eye may be obtained based on the immediately preceding received XR image. Similarly, when the currently received XR image corresponds to the position of the first eye, the second XR image for the second eye may be obtained based on the immediately preceding received XR image.

[0146] Therefore, in each frame, the rendered XR image corresponds to either the position of the first eye or the position of the second eye, and does not need to be reprojected in order to obtain the first XR image or the second XR image, respectively (i.e., until reprojection that takes into account the updated position of the first or second eye).

[0147] The second XR image or the first XR image may also be obtained by reprojecting the drawn XR image received by the XR device in the previous frame, which corresponds to the position of the second eye or the position of the first eye.

[0148] Alternatively, the second or first XR image may be obtained by reprojecting the drawn XR image received in the current frame onto the position of the second eye or the position of the first eye, respectively. Optionally, the reprojected XR image may be modified based on the drawn XR image received in the previous frame.

[0149] When the received XR image corresponds to a position different from both the position of the first eye and the position of the second eye, the received XR image is reprojected by the XR device 12 to both the position of the first eye and the position of the second eye, thereby obtaining the first XR image and the second XR image, respectively.

[0150] In the above, the position of the first eye may be the updated position of the first eye, which may be determined based on the updated position of the XR device 12 when the XR image is displayed.

[0151] Similarly, the position of the second eye may be the updated position of the second eye, which may be determined based on the updated position of the XR device 12 when the XR image is displayed.

[0152] Therefore, the reprojection of the XR image to the position of the first eye and / or the position of the second eye may take into account the movement of the user or the XR device 12 so that the reprojected XR image corresponds to the updated position of the first eye and / or the updated position of the second eye.

[0153] The updated position of the XR device 12 may be determined by the position sensor 20 or by any of the other modifications described above.

[0154] Optionally, the first XR image and / or the second XR image may be corrected by the XR device 12 based on difference data received from at least one XR application instance (step S8).

[0155] The first XR image and the second XR image are displayed by the XR device 12 (step S9).

[0156] In fact, the first XR image may be displayed in part on the projection plane 16 assigned to the first eye, while the second XR image may be displayed in part on the projection plane assigned to the second eye.

[0157] When the first XR image and / or the second XR image are corrected based on difference data, the corresponding corrected first XR image and / or the corrected second XR image may be displayed.

[0158] Certain embodiments disclosed herein, in particular each module and / or unit, utilize circuits (e.g., one or more circuits) to implement the standards, protocols, methodologies, or techniques disclosed herein, to operably connect two or more components, generate information, process information, analyze information, generate signals, encode / decode signals, convert signals, transmit and / or receive signals, control other devices, etc. Any type of circuit may be used.

[0159] In embodiments, the circuit may include one or more computing devices, such as processors (e.g., microprocessors), central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), etc., or any combination thereof, and may include individual digital or analog circuit elements or electronic devices, or combinations thereof. In embodiments, the circuit may include implementations of hardware circuits (e.g., implementations in analog circuits, implementations in digital circuits, etc., and combinations thereof).

[0160] In embodiments, the circuit includes a combination of the circuit and a computer program product having software instructions or firmware instructions stored in one or more computer-readable memories, which cooperate to cause a device to execute one or more protocols, methodologies, or techniques described herein. In embodiments, the circuit includes, for example, a microprocessor or a part of a microprocessor that requires software, firmware, etc., for operation. In embodiments, the circuit includes one or more processors or parts thereof, and associated software, firmware, hardware, etc.

[0161] This application may refer to quantities and figures. Unless otherwise specified, such quantities and figures are not limiting but should be considered examples of quantities or figures that may be associated with this application. In this regard, this application may use the term “plural” when referring to quantities or figures. In this regard, the term “plural” means numbers greater than 1, such as 2, 3, 4, 5, etc. The terms “about,” “approximately,” “nearly,” etc., mean ±5% of the stated value.

Claims

1. An XR distribution method for distributing XR images from an Extended Reality (XR) application instance to an XR device (12), The XR application instance (26) is incorporated into a computer device (14) provided separately from the XR device (12). The aforementioned XR distribution method is The XR application instance (26) draws an XR image for display on the XR device (12) relative to a predetermined reference position, wherein the predetermined reference position has a predetermined relationship with the instantaneous position of the XR device (12). The XR application instance (26) delivers the drawn XR image to the XR device (12), and The steps of reprojecting the drawn XR image to the position of the first eye and / or reprojecting the drawn XR image to the position of the second eye by the XR device (12) so that a first XR image for the first eye and a second XR image for the second eye are obtained based on the drawn XR image, including, XR streaming method.

2. The predetermined reference position is the instantaneous position of the first eye or the instantaneous position of the second eye. The XR distribution method according to claim 1.

3. The predetermined reference position alternately switches between the instantaneous position of the first eye and the instantaneous position of the second eye. The XR distribution method according to claim 2.

4. The predetermined reference position alternates between the instantaneous position of the first eye and the instantaneous position of the second eye for each consecutive drawn XR image. The XR distribution method according to claim 3.

5. When the current drawn XR image corresponds to the position of the second eye, the first XR image for the first eye is obtained based on the previous drawn XR image, and when the current drawn XR image corresponds to the position of the first eye, the second XR image for the second eye is obtained based on the previous drawn XR image. The XR distribution method according to claim 3 or 4.

6. The predetermined reference position is equal to the position of the first eye or the position of the second eye, and the XR distribution method further includes, The XR application instance (26) draws the XR image to be displayed at the position of the second eye or the position of the first eye, thereby obtaining a comparison XR image. The XR application instance (26) reprojects the drawn XR image onto the position of the second eye or the position of the first eye, thereby obtaining the reprojected XR image. The XR application instance (26) performs the steps of comparing the reprojected XR image with the comparison XR image and thereby obtaining difference data, and The XR application instance (26) transmits the difference data to the XR device (12). including, An XR distribution method according to any one of claims 1 to 5.

7. The second XR image for the second eye acquired by the XR device (12) is modified by the XR device (12) based on the difference data, or the first XR image for the first eye acquired by the XR device (12) is modified by the XR device (12) based on the difference data. The XR distribution method according to claim 6.

8. The difference data is encoded by the XR application instance (26) before being transmitted to the XR device (12). The XR distribution method according to claim 6 or 7.

9. The predetermined reference position is different from the position of the first eye, and different from the position of the second eye, The aforementioned XR distribution method further, The XR application instance (26) draws the XR image to be displayed at the position of the first eye, thereby obtaining a first comparison XR image. The XR application instance (26) draws the XR image to be displayed at the position of the second eye, thereby obtaining a second comparison XR image. The XR application instance (26) reprojects the drawn XR image onto the position of the first eye, thereby obtaining a first reprojected XR image. The XR application instance (26) reprojects the drawn XR image onto the position of the second eye, thereby obtaining a second reprojected XR image. The XR application instance (26) compares the first reprojected XR image with the first comparison XR image and thereby obtains first difference data. The XR application instance (26) performs the steps of comparing the second reprojected XR image with the second comparison XR image and thereby obtaining second difference data, and The XR application instance (26) transmits the difference data to the XR device (12). including, The XR distribution method according to claim 1.

10. The first XR image for the first eye acquired by the XR device (12) is modified by the XR device (12) based on the first difference data, and / or the second XR image for the second eye acquired by the XR device (12) is modified by the XR device (12) based on the second difference data. The XR distribution method according to claim 9.

11. The first difference data is encoded by the XR application instance (26) before being transmitted to the XR device (12), and / or the second difference data is encoded by the XR application instance (26) before being transmitted to the XR device (12). The XR distribution method according to claim 9 or 10.

12. The XR application instance (26) and the XR device (12) use the same reprojection algorithm and / or the same machine learning model trained to perform the reprojection. An XR distribution method according to any one of claims 6 to 11.

13. The rendered XR image is reprojected based on the updated position of the XR device (12). An XR distribution method according to any one of claims 1 to 12.

14. XR projection system, The XR projection system (10) comprises an XR device (12) and an XR application instance (26). The XR application instance (26) is incorporated into a computer device (14) provided separately from the XR device (12). The XR projection system (10) is configured to perform any of the XR distribution methods described in claims 1 to 13. XR projection system.

15. The computer device (14) is at least one server or includes at least one server. The XR projection system according to claim 14.

Images