Physical AI device and space operation support method for fully local space use

Abstract

In the operation of spacecraft, lunar rover, space habitation modules, and space robots, equipment status, work environment, worker status, and communication status are handled in a dispersed manner, making it difficult to provide consistent support in the event of communication delays or interruptions. [Solution] A fully local physical AI device for space that acquires equipment status data, environmental images, worker images / voices, and communication status data, locally generates, stores, and references mission-related information including observation events, target objects, work processes, safety constraints, estimated states, recommended actions, and candidate commands to be executed, and generates and displays natural language messages and machine-readable commands, and outputs them via voice output or to a control interface, without relying on a ground-based cloud.

Description

Technical Field

[0001] The present invention relates to state monitoring and operation support in spacecraft, lunar vehicles, planetary exploration vehicles, space habitation modules, space robots, in-ship inspection robots, out-of-ship work support systems, spacesuits, and other space systems. More specifically, it integrates multi-modal observation data including equipment status, working environment, the status of passengers or workers, and communication status, and uses local AI without using the cloud to generate risk assessment, recommended actions, execution candidate commands, and the reasons therefor. The present invention relates to a fully local space physical AI device and a space operation support method.

Background Art

[0002] Conventionally, in the space field, monitoring technologies and support technologies for individual purposes have been proposed, such as terrain recognition on the lunar surface or planetary surface, environmental monitoring inside spacecraft, grasping the status of passengers, and remote operation support for space robots. However, many of these are premised on analysis by ground-based operators or external computing resources, and it is difficult to continue consistent judgment support when communication delays, communication interruptions, bandwidth constraints, or changes in the operating environment occur. In addition, in the prior art, various observation data obtained from vehicles, habitation modules, robots, spacesuits, life support systems, etc. are often handled separately for each individual monitoring system, and a configuration that can understand equipment abnormalities, work processes, worker fatigue, communication status deterioration, danger signs in the working environment, etc. in association with each other and explain the causal relationship or chain of reasons is not sufficient. Furthermore, due to the recent progress of edge AI, technologies for performing image recognition, speech recognition, state estimation, and natural language generation on small computers are known. However, in space systems, while maintaining the relationship between observation events, target objects, work processes, safety constraints, estimated states, recommended actions, and execution candidate commands locally, and taking into account changes in the communication status, a fully local type of physical AI device that generates both explanations in natural language and machine-readable commands has not been fully realized.

Prior Art Documents

[0003] [Patent Document 1] Patent Document 1: Japanese Unexamined Patent Publication No. 2002-257690 [Patent Document 2] Japanese Patent Publication No. 2009-301367 [Non-patent literature]

[0004] [Non-Patent Document 1] Non-Patent Document 1Boyu Kuang, Chengzhen Gu, Zeeshan A. Rana, Yifan Zhao, Shuang Sun, Somtochukwu Godfrey Nnabuife, “Semantic Terrain Segmentation in the Navigation Vision of Planetary Rovers,” Sensors, 2022, 22(21), 8393. [Non-Patent Document 2] T. Vaquero et al., “System Architecture and Operations Concept for NASA's Astrobee Free Flyer,” AIAA SPACE Forum, 2017. Non-patent document 3 Anais Halin, Jacques G. Verly, Marc Van Droogenbroeck, “Survey and Synthesis of State of the Art in Driver Monitoring,” Sensors, 2021, 21(16), 5558. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The object of the present invention is to provide a fully local physical AI device for space and a space operation support method that integrates multimodal observation data acquired with respect to at least one of the following: a spacecraft, lunar rover, space habitation module, space robot, spacesuit, etc., and generates risk levels, operational continuity status, recommended actions, candidate commands to be executed, and the reasons for those actions as consistent natural language messages and machine-readable commands, without relying on the cloud, even in environments where communication delays or communication interruptions may occur. [Means for solving the problem]

[0006] A fully local space physical AI device according to one aspect of the present invention comprises: observation data acquisition means for acquiring observation data including at least one of equipment status data of an observed object, work environment images, images or voices of occupants or workers, and communication status data; relationship information generation means for generating mission relationship information based on the observation data, which includes at least observation events, target objects, work processes, safety constraints, estimated states, recommended actions, and candidate execution commands, and which represents the relationships between these elements; relationship information reference means for storing the mission relationship information in local storage and referencing the mission relationship information corresponding to the current observation data; local AI inference means for generating risk levels, feasibility of continuing operations, recommended actions, reasons for the recommended actions, or candidate execution commands as natural language messages and / or machine-readable commands based on the reference results; output means; and control means. In a preferred embodiment, the observation data acquisition means acquires communication status data indicating at least one of communication delay, communication interruption, bandwidth degradation, packet loss, or link availability, and the local AI inference means switches to at least one of normal support mode, communication degradation support mode, safe hold mode, or autonomous support enhancement mode according to the communication status data and performs inference. In a preferred embodiment, the observation data acquisition means includes environmental imaging means for acquiring terrain, obstacles, visibility or dust conditions on the lunar or planetary surface; crew state acquisition means for acquiring the gaze, posture, movements, speech or biometric indicators of a occupant or worker; and at least one instrument state acquisition means for acquiring at least one of power, heat, pressure, oxygen concentration, carbon dioxide concentration, posture, position, velocity, battery status, manipulator load or spacesuit telemetry. In a preferred embodiment, the relational information referencing means generates a mission status descriptor corresponding to the current observation data, searches for a chain of past observation events or supporting information similar to the descriptor, and the local AI inference means adjusts the priority of descriptive expressions, warning thresholds, or candidate commands based on the search results. In a preferred embodiment, the output means outputs natural language messages to at least one of a display device, a head-mounted display device, an in-spacesuit voice output device, or an in-space voice output device, and outputs machine-readable commands as control candidates for at least one of a lunar vehicle, a space robot, a manipulator, an inspection drone, a life support system, or an alarm system, which are to be executed as necessary after human approval or under predetermined safety constraints. [Effects of the Invention]

[0007] According to the present invention, the acquisition of observational data, generation, storage, and retrieval of related information, inference, explanation generation, and control candidate generation can be completed within a closed local computing system within the space system without relying on constant communication with a ground-based cloud server or external computing resources. Therefore, it is less susceptible to the effects of communication delays, communication interruptions, or bandwidth constraints, and the continuity and safety of space operations can be improved. Furthermore, according to the present invention, since mission-related information is used that correlates equipment status, work environment, passenger or worker status, communication status, and work process, it is possible to explain the reasons for recommended actions or candidate commands along with a chain of justifications, rather than simply outputting alarms, making it easier for users to understand the situation. Furthermore, the present invention can be applied across a wide range of space systems, including lunar rover vehicles, space habitation modules, space robots, in-vehicle inspection robots, extravehicular activity support systems, and spacesuits, making it useful as a platform technology for space-based physical AI centered on fully local AI. [Brief explanation of the drawing]

[0008] [Figure 1] Device Configuration Block Diagram [Figure 2] Processing flow for space operation support methods [Figure 3] Explanation based on mission-related information; schematic diagram of command candidate generation. [Figure 4] Examples of applications of physical AI for space use [Modes for carrying out the invention]

[0009] As shown in Figure 1, the fully local space physical AI device 100 of the present invention comprises an observation data acquisition unit 110, a relational information generation unit 120, a relational information storage unit 130, a relational information reference unit 140, a local AI inference unit 150, an output unit 160, a control interface unit 170, a control unit 180, a power supply unit 190, and a housing 200. Each of these units may be executed on a single edge device, or on multiple computing resources interconnected within a closed local computing system in the same spacecraft. The observation data acquisition unit 110 acquires various observation data according to the space system. For example, when applied to a lunar vehicle or planetary rover, it acquires forward terrain images, potential obstacles, drivable area, attitude, position, speed, drive system temperature, battery status, occupant images, occupant voice, and communication quality. When applied to a space habitation module, it acquires interior images, interior voice, occupant attitude and line of sight, atmospheric pressure, oxygen concentration, carbon dioxide concentration, temperature and humidity, alarm history, door or hatch status, and communication quality. When applied to a space robot or manipulator, it acquires joint angles, load, work object images, attitude, contact status, operator voice, work process information, and communication quality. The relationship information generation unit 120 generates mission relationship information based on the observation data. The mission relationship information includes observation events, target objects, work processes, safety constraints, estimated states, recommended actions, and execution candidate commands as elements, and includes relationships representing the relevance between the elements. For example, "increase in step of the front terrain", "increase in communication delay", "scattered line of sight of passengers", "safe speed constraint", "speed reduction", "autonomous path following candidate", etc. may be associated and recorded. The mission relationship information is stored in the local storage in a graph structure, table structure, key-value structure, JSON format, JSONL format, or other structured formats. The relationship information storage unit 130 may hold the time-series observation history, inference history, approval history, execution history, and anomaly history in association. The relationship information reference unit 140 generates a mission situation descriptor based on the current observation data, and searches for past observation events, chains of basis information, or approved execution histories similar to the descriptor. Thereby, recommended actions or safety constraints that were effective in the past can be reflected in the current situation. The local AI inference unit 150 generates a risk level, operation continuation feasibility, recommended action, reason for the recommended action, or execution candidate command based on the current observation data, mission relationship information, profiles of passengers or workers, work process information, and communication status. The local AI inference unit 150 may include a small language model, an image recognition model, an audio recognition model, a time-series analysis model, an anomaly detection model, or a combination thereof. Preferably, the small language model is fine-tuned based on the space operation manual, inspection procedure manual, work procedure, ground test log, simulation operation log, or past mission data. When the communication status data falls below a predetermined threshold, the local AI inference unit 150 switches from the normal support mode to the communication degradation support mode or the safe hold mode. For example, when the communication delay with the ground increases and the risk level of the front terrain is high, it generates recommended actions or execution candidate commands such as "autonomously move at a reduced speed to a stop candidate point", "hold the manipulator until approval by a person is obtained", and "interrupt the out-of-ship operation and prioritize the return route". The output unit 160 outputs the generated natural language message to an in-ship display device, a head-mounted display device, an in-space-suit voice output device, a speaker, etc. The control interface unit 170 sends the execution candidate command to the lunar vehicle control system, the space robot control system, the manipulator control system, the life support assistance device, the alarm device, and other devices. Preferably, the execution candidate command is limitedly executed after a safety constraint check or is executed after approval by a passenger, an operator, or a ground operator. The control unit 180 integrally controls the above-mentioned units and manages the observation data acquisition cycle, the inference cycle, the display update cycle, the voice output control, and the command transmission control. The power supply unit 190 may be powered from the main power supply, the auxiliary power supply, the battery of the spacecraft or the lunar vehicle, or the in-space-suit power supply. The housing 200 may be configured considering vibration resistance, dust resistance, heat resistance, or radiation resistance.

Example

[0010] (Application example to a lunar vehicle) When the present invention is applied to a lunar vehicle, the environmental imaging means acquires forward terrain images and signs of reduced visibility, the occupant status acquisition means acquires the occupant's line of sight and attitude, and the equipment status acquisition means acquires vehicle speed, attitude, battery temperature, and communication status. The relationship information generation unit 120 associates these observation results as observation events, target objects, driving processes, safety constraints, and recommended actions. If the communication status deteriorates and the terrain unevenness ahead increases, the local AI inference unit 150 generates a natural language message such as "Increased unevenness ahead and increased communication delay are occurring simultaneously. Reduce speed and move to a safe route," and candidate execution commands such as "Switch to route following mode" and "Reduce maximum speed limit." (Example of application to space habitation modules) When the present invention is applied to a space habitation module, the observation data acquisition unit 110 acquires interior images, crew voices, crew posture, oxygen concentration, carbon dioxide concentration, temperature and humidity, hatch status, and communication status. If a crew member falls to the floor, their response is weak, and the carbon dioxide concentration is trending upward, the local AI inference unit 150 outputs a natural language message, "Crew member's fall posture and decreased response detected. Please also check the interior environmental values," and generates candidate approach confirmation commands for the interior alarm system or inspection robot. (Examples of application to space robots or extravehicular activity support) When the present invention is applied to space robots or extravehicular activity support, the observation data acquisition unit 110 acquires images of the work object, the position of tools or bolts, manipulator load, worker voice, spacesuit telemetry, and communication status. If the worker's heart rate load or speech tension increases and the manipulator load approaches an acceptable value, the local AI inference unit 150 outputs a natural language message such as "Worker load and arm load are increasing. Interrupt the current process and reconfirm your posture," and generates a candidate for a manipulator hold command or a safe posture transition command. (Example of referencing mission-related information) As shown in Figure 3, if the current situation descriptor is "high terrain hazard, degraded communication, decreased passenger attention," the related information reference unit 140 searches for similar past observation events, the safety constraints, and approved commands associated with them. The local AI inference unit 150 can reflect these search results and adjust the wording of the explanatory text, the priority of warnings, and the order of command candidates. [Industrial applicability]

[0011] This invention enables observation, inference, explanation, and control candidate generation to be completed within a closed local computing system in space systems that are less dependent on communication with the ground. Therefore, it can be widely used in fields such as lunar exploration, planetary exploration, space habitation, onboard safety management, extravehicular activity support, space robot operation, and spacesuit support. Furthermore, since the present invention can be applied to exploration vehicles on the lunar or planetary surface, manned or unmanned mobile units, space habitation modules, free-flying inspection robots, fixed or movable manipulators, spacesuit support systems, etc., it is extremely useful industrially as a cross-cutting platform technology for physical AI in space. [Explanation of Symbols]

[0012] 100 Fully Local Space-Use Physical AI Devices 110 Observation data acquisition unit 120 Related Information Generation Unit 130 Related Information Storage Unit 140 Related Information Reference Section 150 Local AI Inference Unit 160 Output section 170 Control Interface Unit 180 Control Unit 190 Power supply section 200 cabinets 210 Environmental imaging means 220 Crew status acquisition means 230 Equipment status acquisition means 240 Communication status acquisition means 250 Display device 260 Audio Output Device 270 Space robot or vehicle control system

Claims

1. A fully localized space physical AI device applicable to at least one of a spacecraft, a space habitation module, a space robot, a space work vehicle, or a spacesuit, An observation data acquisition means for acquiring observation data that includes at least one of the following: equipment status data of the observed object, work environment images, images or audio of passengers or workers, and communication status data; A relationship information generation means generates mission relationship information that includes at least observed events, target objects, work processes, safety constraints, estimated states, recommended actions, and candidate commands to be executed, and represents the relationships between these elements, based on the aforementioned observation data. A relational information referencing means that stores the aforementioned mission-related information in local storage and refers to the aforementioned mission-related information corresponding to the current observation data, A local AI inference means generates a risk level, whether operations can be continued, recommended actions, reasons for such recommended actions, or candidate commands to be executed as natural language messages and / or machine-readable commands based on the aforementioned reference results, Output means for displaying or outputting the natural language message and outputting the machine-readable command to the control interface, Control means that executes each of the above means within the spacecraft or within a closed local computing system attached to the spacecraft without communicating with a ground-side cloud server or external computing resources, A fully localized physical AI device for space use, characterized by having the following features.

2. The observation data acquisition means acquires communication status data indicating at least one of communication delay, communication interruption, bandwidth degradation, packet loss, or link availability. The local AI inference means switches to at least one of the following modes depending on the communication status data: normal support mode, communication degradation support mode, safe hold mode, or autonomous support enhancement mode, to generate the recommended action or the candidate execution command. A fully local space-use physical AI device as described in claim 1.

3. The aforementioned observation data acquisition means is Environmental imaging means for acquiring terrain, obstacles, visibility, or dust conditions on the lunar or planetary surface. Crew status acquisition means for acquiring the gaze, posture, movements, speech, or biometric indicators of passengers or workers. Equipment status acquisition means for acquiring at least one of power, heat, pressure, oxygen concentration, carbon dioxide concentration, attitude, position, velocity, battery status, manipulator load, or spacesuit telemetry. including at least one of the following: A fully local space physical AI device according to claim 1 or 2.

4. The aforementioned relational information referencing means generates a mission status descriptor corresponding to the current observation data, searches for a chain of past observation events or supporting information similar to the mission status descriptor, The local AI inference means adjusts the descriptive expression of the natural language message, the warning threshold, or the priority of the candidate execution command based on the search results. A fully local space-use physical AI device as described in claim 1.

5. The output means outputs the natural language message to at least one of a display device, a head-mounted display device, a spacesuit audio output device, or an in-vehicle audio output device, The machine-readable command is output as a control candidate for at least one of the lunar rover, space robot, manipulator, inspection drone, life support system, or alarm system. If necessary, it will be limited to execution after human approval or under specified safety constraints. A fully local space-use physical AI device as described in claim 1.

6. A fully local space operations support method applicable to at least one of a spacecraft, a space habitation module, a space robot, a space work vehicle, or a spacesuit, A step of acquiring observation data that includes at least one of the following: equipment status data, work environment images, images or audio of passengers or workers, and communication status data. Based on the aforementioned observation data, the process generates and locally stores mission-related information including at least the observed event, target object, work process, safety constraints, estimated state, recommended action, and candidate command to be executed. The process involves referencing the mission-related information corresponding to the current observation data to generate a risk level, whether operations can be continued, recommended actions, the reasons for those recommended actions, or candidate commands to be executed as natural language messages and / or machine-readable commands. The process includes displaying or outputting the generated natural language message, and outputting the generated machine-readable command to a control interface. Includes, A fully local space operation support method characterized in that each of the aforementioned steps is performed within the spacecraft or within a closed local computing system associated with the spacecraft, without communication with a ground-based cloud server or external computing resources.

7. A program for causing a computer to perform each step of the method described in claim 6.

8. A non-temporary computer-readable recording medium having the program described in claim 7 recorded on it.

Images