Method and system for providing spatial augmented reality services on earth and earth augmented reality services on spatial platform

By collecting and processing real-time data on the space platform and the earth, generating three-dimensional extended reality environment and avatars, the problem that traditional communication systems cannot effectively transmit human factors is solved, high-quality spatial extended reality services are achieved, and people-to-person communication is enhanced effect.

CN119072674APending Publication Date: 2024-12-03THALES ALENIA SPACE ITALIA SPA CON UNICO SOCIO +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380021976.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-17
Filing Date
2023-02-17
Publication Date
2024-12-03

Smart Images

  • Figure CN119072674A_ABST
    Figure CN119072674A_ABST
Patent Text Reader

Abstract

The present application relates to a method of providing spatial augmented reality services on the earth, comprising: acquiring real-time data relating to a surrounding spatial environment and / or one or more astronauts (3) in said spatial environment by means of one or more acquisition systems / devices (1) mounted on a spatial platform (2); generating, by means of a computer graphics processing device / system (4), a three-dimensional augmented reality environment reproducing the spatial environment and a three-dimensional avatar of the one or more astronauts (3) reproducing the motion and / or action and / or facial expression and / or speech of the one or more astronauts (3) on the basis of the acquired real-time data and the synthetic data, wherein the synthetic data digitally represents a spatial environment and / or one or more astronauts (3); and providing spatial augmented reality services to one or more users (5) on the earth based on the generated three-dimensional augmented reality environment and avatar.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to European patent application No. 22425006.8 filed on February 17, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to mixed and virtual reality technologies, and more generally, extended reality (XR) technologies, which are exploited through the use of computer graphics in conjunction with digital modeling techniques, such as those used by digital twin solutions.

[0004] More specifically, the present application relates to a method and system for providing a space-extended reality service on Earth and a method and system for providing an Earth-extended reality service on a space platform. Background Art

[0005] As is known, the systems and devices currently used in the technical field of communication between people on Earth and astronauts on space missions are mainly based on traditional audio and video technologies. In particular, the audio communication systems currently used in the space industry are monophonic and only in very rare cases stereophonic. In addition, communication based on visual content is mainly achieved by means of two-dimensional (2D) systems and devices (mainly displays / monitors / screens that display 2D images, videos, photos, charts, written text, etc.).

[0006] Although the users of these systems (i.e., astronauts and ground control personnel) are accustomed to communicating in 2D, it is absolutely clear that face-to-face communication can convey a great deal of information and emotions that cannot be transmitted by the audio / visual media currently used in the space industry. This undoubtedly represents a limitation worth overcoming, especially (but not only) in terms of safety aspects and activities to be monitored and controlled remotely during space exploration missions.

[0007] Current space communication channels are sufficient for transmitting telemetry data and, more generally, binary information, but for long-duration manned space flights, where the human factor is fundamentally crucial, "wider" channels become a necessity for human-to-human communication, since the human factor represents the weakest link in the entire chain leading to the success of long-duration manned flights. In fact, it is generally known and widely documented in the scientific literature that poor communication is the most frequent cause of accidents in the space industry (but not only), so that disciplines such as cognitive ergonomics aim to solve this problem. Summary of the invention

[0008] It is an object of the present application to provide a system that overcomes the known drawbacks of current systems for space-to-Earth and Earth-to-space communications.

[0009] This and other objects are achieved by the present application in that the present application, as defined in the appended claims, relates to a method and system for providing a spatial extended reality service on Earth and a method and system for providing an Earth extended reality service on a space platform.

[0010] In particular, the method for providing a spatial extended reality service on Earth according to the present application includes:

[0011] a) collecting real-time data related to the surrounding space environment and / or one or more astronauts in the space environment by means of one or more collection systems / devices installed on the space platform;

[0012] b) generating, with the aid of a computer graphics processing device / system, a three-dimensional (3D) extended reality environment reproducing the space environment and one or more astronaut 3D avatars reproducing the movements and / or actions and / or facial expressions and / or voices of the astronaut based on the acquired real-time data and synthetic data, wherein the synthetic data digitally represents the space environment and / or the astronaut; and

[0013] c) providing spatial extended reality services to one or more users on Earth based on the generated 3D extended reality environment and avatar.

[0014] In addition, the method for providing earth extended reality service on a space platform according to the present application includes:

[0015] a) collecting real-time data related to the surrounding ground environment and / or one or more ground users in the ground environment by means of one or more collection systems / devices installed on the earth;

[0016] b) generating, by means of a computer graphics processing device / system, a 3D extended reality environment reproducing a ground environment and one or more ground user 3D avatars reproducing the movements and / or actions and / or facial expressions and / or voices of the ground users based on the collected real-time data and synthetic data, wherein the synthetic data digitally represents the ground environment and / or the ground users; and

[0017] c) providing Earth extended reality services to one or more astronauts on the space platform based on the generated 3D extended reality environment and avatar. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] For a better understanding of the present application, preferred embodiments purely as non-limiting examples will now be described with reference to the accompanying drawings (not drawn to scale) which schematically illustrate an example of a high-level system architecture according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following discussion is presented to enable those skilled in the art to understand, make and use the present application. Various modifications to the embodiments will be readily apparent to those skilled in the art without departing from the scope of protection claimed in the present application. Therefore, the present application is not intended to be limited to the embodiments shown and described, but is consistent with the broadest scope of protection consistent with the principles and features disclosed herein and defined in the appended claims.

[0020] The present application relates to a method for providing a spatially extended reality service on Earth, the method comprising:

[0021] a) collecting real-time data related to the surrounding space environment and / or one or more astronauts in the space environment by means of one or more collection systems / devices installed on a space platform (e.g., a space module, a space station, a space vehicle, a spacecraft, etc.);

[0022] b) generating, with the aid of a computer graphics processing device / system, a three-dimensional (3D) extended reality environment reproducing the space environment and one or more astronaut 3D avatars reproducing the movements and / or actions and / or facial expressions and / or voices of the astronaut based on the acquired real-time data and synthetic data, wherein the synthetic data digitally represents the space environment and / or the astronaut; and

[0023] c) providing spatial extended reality services to one or more users on Earth based on the generated 3D extended reality environment and avatar.

[0024] Preferably, the real-time data includes data such as a point cloud digitally representing the astronaut's body position and / or body posture and / or facial expression.

[0025] Conveniently, the real-time data comprises images and / or photographs and / or video data and / or audio data.

[0026] Preferably, synthetic data digitally representing the space environment and / or one or more astronauts is generated based on one or more computer-aided design models and / or one or more digital twin models.

[0027] Conveniently, one or more users experience the spatial extended reality service using one or more extended reality devices.

[0028] The present application also relates to a spatially extended reality service provided on Earth by implementing the aforementioned method, and to a system designed to provide the spatially extended reality service on Earth, wherein the system comprises:

[0029] One or more acquisition systems / devices mounted on the space platform and configured to acquire real-time data related to the surrounding space environment and / or one or more astronauts in the space environment; and

[0030] A computer graphics processing device / system, installed on the earth or on a space platform, or using a distributed architecture, partially installed on the earth and partially installed on a space platform, and configured to receive the collected real-time data and perform steps b) and c) of the method as described above.

[0031] In addition, the present application also relates to a method for providing earth extended reality service on a space platform, the method comprising:

[0032] a) collecting real-time data related to the surrounding ground environment and / or one or more ground users in the ground environment by means of one or more collection systems / devices installed on the earth;

[0033] b) generating, by means of a computer graphics processing device / system, a 3D extended reality environment reproducing a ground environment and one or more ground user 3D avatars reproducing the movements and / or actions and / or facial expressions and / or voices of the ground users based on the collected real-time data and synthetic data, wherein the synthetic data digitally represents the ground environment and / or the ground users; and

[0034] c) providing earth extended reality services to one or more astronauts on the space platform based on the generated 3D extended reality environment and one or more avatars.

[0035] Preferably, the real-time data comprises data, such as a point cloud, digitally representing body positions and / or body postures and / or facial expressions of one or more ground users.

[0036] Conveniently, the real-time data comprises images and / or photographs and / or video data and / or audio data.

[0037] Preferably, synthetic data digitally representing the ground environment and / or ground users is generated based on one or more computer-aided design models and / or one or more digital twin models.

[0038] Conveniently, one or more astronauts use one or more extended reality devices to experience the Earth extended reality service.

[0039] The present application also relates to an earth extended reality service provided on a space platform by implementing the aforementioned method, and to a system designed to provide the earth extended reality service on a space platform, wherein the system comprises:

[0040] One or more acquisition systems / devices installed on the earth and configured to acquire real-time data related to the surrounding ground environment and / or one or more ground users in the ground environment; and

[0041] A computer graphics processing device / system, installed on the earth or on a space platform, or using a distributed architecture, partially installed on the earth and partially installed on a space platform, and configured to receive the collected real-time data and perform steps b) and c) of the method as described above.

[0042] The present application originates from the idea of ​​the applicant to utilize the mixing of real-time data generated in external space (e.g., images, video, audio, data related to the facial expressions of astronauts, etc.) with synthetic data generated by computer graphics applications such as computer-generated imagery (CGI) programs (e.g., computer-aided design (CAD) models, terrain maps, digital twin systems / models, etc.) to achieve their real-time conversion and utilization for the purpose of a 3D immersive environment on Earth. Obviously, the opposite scenario (i.e., creating a 3D immersive environment in space that reproduces places and people on the ground) is also possible.

[0043] The steps of the process according to this application can be conveniently divided into the following main activities:

[0044] · In a spatial environment, capture / generate / produce / collect real-time data, in particular real-time images and / or photos and / or videos and / or audio and / or point clouds;

[0045] Process real-time data and transmit it to the ground;

[0046] Referencing the received data to predefined synthetic data of the space environment available on Earth (e.g., a digital twin of a space module / vehicle / outpost); and

[0047] · Virtually reproduce the space environment and the activities performed by astronauts in said space environment in real time on Earth.

[0048] The communications system installed on the currently inhabited space module provides monitoring, management and control of the onboard audio and video channels (as well as data from the onboard data management system), and their transmission to the space control center on Earth.

[0049] Conveniently, in addition to the conventional communication systems on board the space vehicle / module / lander, additional devices can be advantageously installed on board that are capable of tracking and collecting additional real-time data related to the space environment and the astronaut's movements and facial expressions, and the additional real-time data is connected to the existing system (e.g., to the onboard communication switching matrix device). The captured signals can then be sent in real time on Earth (or anywhere useful) and carefully crafted to recreate the captured remote environment by "mixing" the real-time data with predefined synthetic data. Thus, the remote extended reality environment generated by combining real-time and synthetic data can be conveniently utilized by using, for example, a peer-to-peer (P2P) communication network (e.g., for ground control remote support) or a broadcast network (e.g., for education, gaming, metaverse, etc. applications).

[0050] The accompanying drawings schematically illustrate examples of high-level system architectures according to embodiments of the present application.

[0051] As shown in the accompanying drawings, the acquisition system / device 1 installed on a space platform 2 (e.g., a space module, a space vehicle, a spacecraft, etc.) is used to acquire real-time data related to the surrounding space environment and / or one or more astronauts 3 in the space environment (e.g., images and / or photos and / or video data and / or audio data and / or data (e.g., a point cloud digitally representing the body position and / or body posture and / or facial expression, etc. of the astronaut 3)).

[0052] The collected real-time data is then sent (for example, with the aid of a conventional onboard telemetry system) to Earth, where a ground-based computer graphics processing device / system 4 uses the received data to create / generate / produce 3D avatars of one or more astronauts 3, reproducing the movements, actions, facial expressions, voice, etc. of the one or more astronauts 3 in a 3D extended reality environment that reproduces the space environment and is created / produced / generated based on synthetic data available on Earth (such as CAD and / or digital twin models).

[0053] The ground computer graphics processing device / system 4 outputs a data stream transmitted to one or more ground users 5 via one or more ground communication systems / networks, and the one or more users 5 use an extended reality (XR) device 6 (e.g., a virtual reality (VR) headset, augmented reality (AR) glasses, etc.) to enjoy a 3D extended reality environment with the avatars of one or more astronauts 3. The one or more ground users 5 thereby experience the same immersive environment as the environment in which the one or more astronauts 3 operate.

[0054] Thus, it is worth noting that in the example shown in the figures, the ground-based computer graphics processing device / system 4 is installed on the earth. However, according to two different embodiments, the computer graphics processing device / system may be installed on the space platform 2, or it may be implemented with a distributed architecture, whereby the computer graphics processing method is partially installed on the space platform 2 and partially installed on the earth.

[0055] The present application can be advantageously used for space-ground human-to-human communications and ground-space human-to-human communications to:

[0056] Providing various services, such as new methodologies for training astronauts or space tourists for future space tourism missions;

[0057] Provide advanced tools for remote technical support services provided by ground control centers;

[0058] Strengthen space entertainment business;

[0059] Improving remote social interactions between astronauts and people on Earth;

[0060] ·etc.

[0061] The present application allows to circumvent the inherent problems / deficiencies of real-time transmission of massive data from space to earth and from earth to space, problems / deficiencies mainly related to bandwidth requirements that cannot be met by current space-to-terrestrial and ground-to-space telecommunication systems.

[0062] The present application overcomes these problems / deficiencies by combining the use of a limited amount of "live" real-time data from space with a massive amount of "synthetic" data from digital twins available on the ground (or using a limited amount of "live" real-time data from the ground with a massive amount of "synthetic" data from digital twins available in space).

[0063] In fact, the combined use of extended reality technology and computer graphics technology along with synthetic data allows for the mitigation of real-time streaming of data from space while still allowing for the re-creation of an immersive extended reality environment on Earth that reproduces a remote space environment (or allows for the mitigation of real-time streaming of data from Earth while still allowing for the re-creation of an immersive extended reality environment on Earth that reproduces an Earth environment).

[0064] In particular, the present application allows to reproduce on the ground the avatars of the astronauts (including their facial expressions) and the entire space environment in which they move with a delay that depends mainly on the physical distance between the Earth and the relevant location in space (e.g., the Earth's orbit, the Moon's orbit, a Mars mission, etc.) and only slightly on the time required to calculate the reconstruction of the space environment in an immersive extended reality scene.

[0065] Thus, the system according to the present application is not only capable of transmitting 2D images of a spatial environment like conventional communication systems currently installed on space platforms, but is also capable of recreating the same environment in real time in an immersive mode so that the end user has the perception of being "there".

[0066] The present application can be advantageously used for many applications such as onboard training and experiments, ground control center operations support, astronaut remote family conferencing, crew leisure activities, reducing crew psychosocial deprivation during long-term space exploration missions, etc. In addition, additional applications may also include video broadcasting, video games, educational / social applications, etc. In addition, national space agencies can benefit from the strong emotional impact of the system to increase public awareness and participation in space-related activities.

[0067] It is noteworthy that the system according to the present application also allows the same concept to be implemented in the opposite communication direction (i.e., from Earth to space). In fact, the same service can allow space crews to enjoy an immersive extended reality ground environment, thereby providing a very valuable psychosocial countermeasure tool for the crew.

[0068] As explained previously, all current communication solutions in the spatial domain utilize only 2D video / images as the visual communication medium, whereas synthetic data such as CAD models are only used to reproduce "offline" 3D objects or VR environments.

[0069] In other words, nowadays, the synthetic materials generated after the design of the space module (such as CAD models) and the telemetry data transmitted by the space module are only used and managed in asynchronous mode, that is, reproducing an "offline" virtual environment to facilitate activities such as engineering or usability testing.

[0070] In contrast, the present application provides for the use of a combination of real-time data collected in space and synthetic data generated on Earth (e.g., digital twins) to virtually reproduce astronaut activities in an immersive virtual space scene in real time on the ground.

[0071] As the space domain, and more specifically as human space exploration, is of particular interest for safety reasons, but not only in terms of communications (telemetry data and interpersonal communications between space crew and personnel in ground control centers), the present application can generate valuable assistance and improvements in human-to-human communications.

[0072] In addition to using this application in a strictly spatial environment, if applied to a wider audience (such as applications for education, entertainment, etc.), the emotional impact perceived by the subject when using the system can amplify user participation in the activity.

[0073] Therefore, the present application has a dual value: on the one hand, improving the quality of communication and interaction between space crew and ground personnel, and on the other hand, being applied to the general public, making topics involving space more fascinating and attractive.

[0074] In fact, the present application is based on a new extended reality functional architecture, which is capable of providing services to different categories of end users, both in the space domain and in non-space, by combining real-time data generated in space facilities and data of 3D digital models of known configurations (i.e., digital twins) involving space infrastructure.

[0075] In summary, the different industrial applications and different commercial applications of the present application can be classified into at least the following four main categories:

[0076] 1) Science related – remote real-time verification of experiments by astronauts using extended reality technology to interact with researchers present from Earth;

[0077] 2) Training – virtual experiential training, even with the direct participation of astronauts in space, live broadcasting their guidance and practical experience: on-site personal trainers;

[0078] 3) Education and entertainment – ​​knowledge promotion and a wealth of information using the most modern XR technologies;

[0079] 4) Commercial – Promotion of space assets and airborne products to the public.

[0080] From the above, it can be seen that the innovative features and technical advantages of the present application are immediately clear to those skilled in the art.

[0081] In summary, it is clear that the present application can be subjected to all various modifications and variations that are within the protective scope of the present application as defined in the appended claims.

Claims

1. A method for providing a spatially extended reality service on Earth, comprising: a) collecting real-time data related to the surrounding space environment and / or one or more astronauts (3) in the space environment by means of one or more collection systems / devices (1) installed on the space platform (2); b) generating, by means of a computer graphics processing device / system (4), a three-dimensional extended reality environment reproducing the space environment and one or more three-dimensional avatars of the one or more astronauts (3) reproducing the movements and / or actions and / or facial expressions and / or voices of the one or more astronauts (3) based on the collected real-time data and synthetic data, wherein the synthetic data digitally represents the space environment and / or the one or more astronauts (3); and c) providing spatial extended reality services to one or more users (5) on Earth based on the generated three-dimensional extended reality environment and one or more avatars.

2. The method according to claim 1, wherein: The real-time data includes data such as point clouds that digitally represent the body position and / or body posture and / or facial expression of the astronaut (3).

3. The method according to claim 1 or 2, wherein: The real-time data includes images and / or photos and / or video data and / or audio data.

4. The method according to any one of claims 1 to 3, wherein: The synthetic data is generated based on one or more computer-aided design models and / or one or more digital twin models that digitally represent the space environment and / or the astronaut (3).

5. A method according to any one of the preceding claims, wherein: The one or more users (5) use one or more extended reality devices (6) to experience the spatial extended reality service.

6. A spatially extended reality service provided on Earth by implementing the method as claimed in any one of the preceding claims.

7. A system designed to provide a spatially extended reality service on Earth, comprising: One or more acquisition systems / devices (1) mounted on a space platform (2) and configured to acquire real-time data related to the surrounding space environment and / or one or more astronauts (3) in said space environment; and A computer graphics processing device / system (4) configured to receive the acquired real-time data and to perform steps b) and c) of the method according to any one of claims 1 to 5.

8. A method for providing an earth extended reality service on a space platform, comprising: a) collecting real-time data related to the surrounding ground environment and / or one or more ground users in the ground environment by means of one or more collection systems / devices installed on the earth; b) generating, by means of a computer graphics processing device / system, a three-dimensional extended reality environment reproducing the ground environment and one or more three-dimensional avatars of the one or more ground users reproducing the movements and / or actions and / or facial expressions and / or voices of the one or more ground users based on the collected real-time data and synthetic data, wherein the synthetic data digitally represents the ground environment and / or the one or more ground users; and c) providing earth extended reality services to one or more astronauts on the space platform based on the generated three-dimensional extended reality environment and one or more avatars.

9. The method according to claim 8, wherein: The real-time data includes data, such as a point cloud, digitally representing body positions and / or body postures and / or facial expressions of the one or more ground users.

10. The method according to claim 8 or 9, wherein: The real-time data includes images and / or photos and / or video data and / or audio data.

11. The method according to any one of claims 8 to 10, wherein: The synthetic data is generated based on one or more computer-aided design models and / or one or more digital twin models digitally representing the ground environment and / or the one or more ground users.

12. The method according to any one of claims 8 to 11, wherein: The one or more astronauts use one or more extended reality devices to experience the Earth extended reality service.

13. An earth extended reality service provided on a space platform by implementing the method according to any one of claims 8 to 12.

14. A system designed to provide an Earth extended reality service on a space platform, comprising: One or more acquisition systems / devices installed on the earth and configured to acquire real-time data related to the surrounding ground environment and / or one or more ground users in the ground environment; and A computer graphics processing device / system configured to receive the acquired real-time data and execute steps b) and c) of the method according to any one of claims 8 to 12.