Method and system for performing telexistence

The system addresses the limitation of single-user telexistance by enabling multi-user collaboration and secure control handover in telexistance systems, utilizing machine learning and limited hardware for effective remote operation and viewer access.

JP2025060410AActive Publication Date: 2025-04-10HITACHI LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024124418
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-09-29
Filing Date
2024-07-31
Publication Date
2025-04-10
Estimated Expiration
2044-07-31

Smart Images

  • Figure 2025060410000001_ABST
    Figure 2025060410000001_ABST
Patent Text Reader

Abstract

To enable a plurality of users to continuously remotely operate / control a humanoid.SOLUTION: A method for performing telexistence according to the present invention includes: receiving, by a processor, a request to control a humanoid from a remote user; determining acceptance of the request by a host; for the request being determined as accepted, verifying, by the processor, receipt of a set of initiating signals; for the set of initiating signals being received: receiving a humanoid mapping method chosen by the remote user, and controlling the humanoid on the basis of the humanoid mapping method determined by the remote user; and for the set of initiating signals not being received, terminating, by the processor, the request to control the humanoid.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to methods and systems for implementing telexistence. [Background technology]

[0002] In the information technology (IT) field, collaboration with people across different parts of the world is now possible using collaboration software. The use of collaboration software allows users to share, review and modify materials in real time. However, such interactive collaboration software or hardware is not available in fields that involve physical interaction. Traditional collaboration methods require participants to be physically present in a single location in real time. Travel to a meeting location tends to incur additional operational costs and cause delays, which can lead to reduced customer satisfaction.

[0003] Telexistence is a concept that allows users to remotely control a humanoid or robot to perform actions with real-world objects, thereby allowing users to interact in physical space. However, collaboration through telexistence for multiple users is limited, since control of a humanoid is limited to a single user at a single event. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-40155 A Summary of the Invention [Problem to be solved by the invention]

[0005] Related art discloses a method for remotely controlling a humanoid by one person through a virtual reality (VR) headset and mobile hardware, but the method does not allow multiple users to remotely operate / control the humanoid in succession in a single function / event, nor does it allow a remotely connected viewer to remotely view the actions of the humanoid.

[0006] Related art discloses a method of controlling a humanoid through a stationary control system. The stationary control system allows the humanoid to be operated by mapping / mimicking the user's actions through a wearable control device of the control system. However, the stationary control system itself can be expensive, and multiple users cannot operate / control the humanoid remotely in succession. [Means for solving the problem]

[0007] One aspect of the disclosure involves an innovative method of implementing telexistence, the method including receiving, by a processor, a request to control a humanoid from a remote user, determining, by the processor, acceptance of the request by the host, if the request is determined to be accepted, verifying, by the processor, receipt of a set of activation signals, if the set of activation signals is received, receiving a humanoid mapping method selected by the remote user and controlling the humanoid based on the humanoid mapping method determined by the remote user, and terminating, by the processor, the request to control the humanoid if the set of activation signals is not received. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example system for a humanoid 100, according to one embodiment. [Diagram 2] FIG. 2 illustrates an example of a remote configuration 200 for a remote user to control a humanoid 100, according to one embodiment. [Diagram 3]FIG. 3 illustrates an example of a remote viewer configuration 300, according to one embodiment. [Figure 4] FIG. 4 illustrates an example of a process flow 400 for implementing telexistence, according to one embodiment. [Diagram 5] FIG. 5 illustrates an example of a process flow 500 for implementing telexistence after a remote user has been handed control of a humanoid 100, according to one embodiment. [Figure 6] FIG. 1 illustrates an example computing environment having an example computing device suitable for use in some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A general architecture for implementing various features of the present disclosure will now be described with reference to the drawings. The drawings and the associated description are provided to illustrate embodiments of the present disclosure and are not intended to limit the scope of the present disclosure. Reference numbers are re-used throughout the drawings to indicate correspondence between referenced elements.

[0010] The following detailed description provides details of the figures and examples of the present application. Reference numbers and descriptions of elements that overlap between figures are omitted for clarity. Terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic implementation or a semi-automatic implementation that requires user or administrator control over certain aspects of the implementation, depending on the desired implementation in the art of practicing the examples of the present application. Selection may be performed by a user through a user interface or other input means, or may be realized through a desired algorithm. The examples as described herein may be utilized alone or in combination, and the functionality of the examples may be realized through any means according to the desired implementation.

[0011] The present embodiment relates to a method and system for implementing multi-user telexistence through multi-user collaboration. The embodiment provides multi-user collaborative telexistence with limited hardware while ensuring a safe subsequent handover of the humanoid by a remote user. The user's body movements are predicted using a camera utilizing machine learning algorithms. The camera can be a built-in camera associated with a computing device such as a laptop, tablet, etc. In some embodiments, an external camera may be used instead of a built-in camera. If the user wishes to control the humanoid, a VR headset would be required to provide additional movement signals to the system.

[0012] FIG. 1 illustrates an example system of a humanoid 100, according to one embodiment. As illustrated in FIG. 1, the humanoid 100 may include components such as a sensing system 101, a control system 102, an actuation system 103, and a camera 104. The sensing system 101 may include components such as, but not limited to, audio sensors, motion sensors, load sensors, touch sensors, and the like. The sensing system 101 may be used to detect combined weight loads and issue combined weight signals representative of the detected combined weight loads. The control system 102 is used to control the behavior / movement of the humanoid 100 and may receive signals from other components of the humanoid 100 and from a remote user to implement control of the humanoid 100. The actuation system 103 implements the movement of the humanoid 100 and may involve at least one of hydraulic actuators, pneumatic actuators, or electric actuators. Control signals are sent from the control system 102 to the actuation system 103 to provide instructions for the movement control. The camera 104 allows a remote user to view the humanoid's environment and actions as perceived from the humanoid's 100 perspective over a communications network.

[0013] In some embodiments, the humanoid 100 may also include a screen display 105. The screen display 105 can be used for viewing or screen sharing of the environment by the user or other viewers / participants. The humanoid 100 replicates the actions of the user remotely from anywhere on Earth. In some embodiments, immediate customer assistance can be provided through the use of the humanoid 100.

[0014] FIG. 2 illustrates an example remote configuration 200 for a remote user to control a humanoid 100, according to one embodiment. The remote configuration 200 communicates with the humanoid 100 through a network, not shown in the drawing. As shown in FIG. 2, the remote configuration 200 may include devices such as, but not limited to, a virtual reality (VR) device 201 and a camera 202. The VR device 201 can be a VR headset that allows the remote user to view the humanoid's environment as perceived through the camera 104 of the humanoid 100. The camera 202 may be an in-built camera of a computing device (e.g., laptop, tablet, smartphone, etc.) or an external camera connected to a computing device (e.g., personal computer, laptop, etc.). The camera 202 is pointed at the remote user and tracks / captures the remote user's body movements. In some embodiments, the camera 202 may send a captured feed over the network for display on the screen display 105 of the humanoid 100.

[0015] In some embodiments, the remote arrangement 200 may further include an optional display device 203, such as a monitor, that allows the remote user to display and view the humanoid's environment as perceived through the humanoid's 100 camera 104. Information received from the humanoid 100, such as warning messages or notifications, may be displayed on the VR equipment 201 or the display device 203.

[0016] Figure 3 illustrates an example remote viewer arrangement 300, according to one embodiment, where the remote viewer is a participant in a humanoid control session in which the humanoid 100 is controlled by a remote user. The remote viewer arrangement 300 communicates with the humanoid 100 through a network, which is not shown in the drawing. As shown in Figure 3, the remote viewer arrangement 300 may include devices such as, but not limited to, a display device 301. Actions performed by the humanoid 100 or the humanoid's environment as perceived through the camera 104 of the humanoid 100 can be visualized on the display device 301.

[0017] In some embodiments, the remote viewer arrangement 300 may further include VR equipment 302 for visualizing actions performed by the humanoid 100 or the humanoid's environment as perceived through the camera 104 of the humanoid 100. In some embodiments, the remote viewer arrangement 300 may further include a camera 303. The camera 303 is similar to the camera 202 of FIG. 2 and may be a built-in camera of a computing device (e.g., laptop, tablet, smartphone, etc.) or an external camera connected to a computing device (e.g., personal computer, laptop, etc.). The camera 303 may transmit a captured feed over a network for display on the screen display 105 of the humanoid 100.

[0018] Either one or more remote viewers may be host-authenticated users. Like remote users, host-authenticated users may also be given access to control the humanoid 100 if they have the necessary devices to control the humanoid 100.

[0019] FIG. 4 shows an example process flow 400 for implementing multi-user telexistence, according to one embodiment. The process begins in step S401, where a request to control the humanoid 100 is sent by a remote user. In step S402, a determination is made whether the host or the host authenticated user has accepted the remote user's request. The right to provide control of the humanoid 100 to the remote user is granted to the host as well as the host authenticated user. If the request is not accepted in step S402, then in step S403, the remote user's request for control is denied.

[0020] In step S404, a movement / input signal is sent from the remote user's VR equipment 201 (e.g., a VR headset). In step S405, a movement detection signal, such as a feed signal, is received from the camera 202. In step S406, a combined weight signal indicative of the combined weight load is transmitted from the sensing system 101 of the humanoid 100. The three signals of steps S404-S406, i.e., the activation signal set, must be received in order for the remote user to receive control of the humanoid 100. In step S407, a verification or determination of receipt of the set of activation signals is performed.

[0021] If no signal of the set of activation signals is received in step S407, control of the humanoid 100 is disallowed in step S408. If a set of activation signals is received in step S407, the process proceeds to step S409, where the remote user selects either humanoid-to-user mapping or user-to-humanoid mapping as the humanoid mapping method. The remote user is given a choice of two ways to reach a common initial pose for himself and the humanoid 100 before taking control of the humanoid 100. The two choices are humanoid-to-user mapping, shown as method 1 in FIG. 4, and user-to-humanoid mapping, shown as method 2 in FIG. 4. In humanoid-to-user mapping, the humanoid 100 reaches / reproduces the remote user's body pose or orientation before handing over control. In user-to-humanoid mapping, the remote user takes over from the humanoid 100 by reaching / reproducing a pose or orientation of the humanoid 100 from the orientation / configuration left over from a previous terminated telexistence session associated with the previous remote user. If, in step S409, the remote user selected user-to-humanoid mapping as the humanoid mapping method, the process continues to step S410 and the remote user is granted control of the humanoid 100.

[0022] If the remote user selects humanoid-to-user mapping as the humanoid mapping method, the process proceeds to step S411 where a determination is made based on the pose or orientation of the humanoid 100 as to whether the combined weight load is significant. A significant combined weight load indicates that the humanoid 100 may be holding a weight / item. The combined weight load is determined from the combined weight signal received from the sensing system 101 of the humanoid 100 in step S406. If the combined weight load is determined to be not significant, the process continues to step S410 and the remote user is allowed to control the humanoid 100.

[0023] If the combined weight load is determined to be significant, a first warning message is sent to the remote user in step S412 alerting the remote user to a potential safety concern that may result from the humanoid-to-user mapping. The first warning message may indicate a potential for a weight / item to be dropped and potential damage that may be caused by the dropped weight / item. In some embodiments, the remote user may receive a request to change the humanoid mapping method as part of the first warning message, which leads to a determination in step S413 whether the remote user agrees to the request to change the humanoid mapping method.

[0024] If, in response to the first warning message, the remote user chooses to change the humanoid mapping method and requests the change in step S413, the user is allowed control of the humanoid 100 in step S410. If the remote user chooses to proceed with humanoid-to-user mapping as the humanoid mapping method, the process continues in step S414, where a second warning message is sent to the remote user to warn of a possible hazard that may occur. Upon issuance of the second warning message, the user is allowed control of the humanoid 100 in step S410. In some embodiments, the remote user is asked to respond to the second warning message, which may be a request for confirmation / reconfirmation of the humanoid mapping method for the remote user. Such a response may include confirmation of the humanoid mapping method by clicking an accept button through an input device. The first and second warning messages are sent to the remote user and displayed on the VR equipment 201 or any display device 203. In step S415, the camera 104 of the humanoid 100 displays the humanoid's environment and the movements of the humanoid 100 on the VR device 201 or any display device 203.

[0025] To pass control of the humanoid 100 from a current remote user to a subsequent remote user, the subsequent remote user must follow the steps shown in Figure 4. When passing control, control of the humanoid 100 by the current remote user, who becomes the previous remote user, is terminated and the subsequent remote user is granted control of the humanoid 100.

[0026] 5 shows an example process flow 500 for implementing telexistence after a remote user is handed control of the humanoid 100, according to one embodiment. The process begins with step S501, where motion / input signals are sent from the remote user's VR equipment 201 (e.g., VR headset), step S502, where a trained machine learning (ML) model generates motion predictions as signals to control the humanoid 100, which are sent to the control system 102, and step S503, where a continued permission to control the host or host-authenticated humanoid 100 is received. In step S502, the remote user's body motions / tracked motions as observed by the camera 202 are fed to the trained ML model to generate motion predictions / motion mappings, and the predicted motions are sent to the control system 102 as signals to control the humanoid 100.

[0027] Signals and permissions as obtained from steps S501-S503 must continue to be received in order for the remote user to maintain control of the humanoid 100. In step S504, the humanoid 100 performs the predicted movement based on the signal received in step S502. The process then continues to step S505, where the camera 104 of the humanoid 100 broadcasts the humanoid's environment and the movements / behavior of the humanoid 100 over the network to the display device 301 or optional VR equipment 302 for viewing by the remote viewer.

[0028] The above-described embodiments may have various benefits and advantages. For example, the embodiments allow instant access to the physical environment for multiple users to collaborate with limited hardware capabilities through multi-user telexistence. The embodiments may be used to perform, but are not limited to, remote security inspection, audits, maintenance, manufacturing, consultations, and the like. Remote maintenance / troubleshooting may provide instant access to support during manufacturing while reducing costs and wait times.

[0029] 6 illustrates an example computing environment having an example computing device suitable for use in some embodiments. The computing device 605 of the computing environment 600 can include one or more processing units, cores, or processors 610, memory 615 (e.g., RAM, ROM, and / or other), internal storage 620 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 625, any of which can be coupled with a communication mechanism or bus 630 for communicating information or embedded in the computing device 605. The IO interface 625 is also configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0030] Computing device 605 can be communicatively coupled to input / user interface 635 and output device / interface 640. Either or both of input / user interface 635 and output device / interface 640 can be wired or wireless interfaces and can be removable. Input / user interface 635 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch screen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, accelerometer, optical reader, and / or others). Output device / interface 640 can include a display, television, monitor, printer, speaker, Braille, etc. In some embodiments, input / user interface 635 and output device / interface 640 can be embedded in or physically coupled to computing device 605. In other embodiments, other computing devices can function as or provide the functionality of input / user interface 635 and output device / interface 640 of computing device 605.

[0031] Examples of computing devices 605 may include, but are not limited to, highly mobile devices (e.g., smart phones, automobiles and other mechanical devices, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with and / or coupled to one or more embedded processors, radios, etc.).

[0032] Computing device 605 may be communicatively coupled (e.g., via IO interface 625) to external storage 645 and network 650 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 605, or any connected computing device, may function, provide services, or be referred to as a server, a client, a thin server, a general purpose machine, a special purpose machine, or another label.

[0033] The IO interface 625 can include, but is not limited to, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 802.11x, Universal Serial Bus, WiMAX, modem, cellular network protocols, etc.) to communicate information to and from at least all connected components, devices, and networks of the computing environment 600. The network 650 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0034] The computing device 605 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0035] The computing device 605 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions can be retrieved from a transitory medium and can be stored in and retrieved from a non-transitory medium. The executable instructions can be in one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0036] The processor 610 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 660, an application programming interface (API) unit 665, an input unit 670, an output unit 675, and an inter-unit communication mechanism 695 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can vary in design, function, configuration, or implementation and are not limited to the description provided. The processor 610 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0037] In some embodiments, once information or instructions to execute are received by API unit 665, they may be communicated to one or more other units (e.g., logic unit 660, input unit 670, output unit 675). In some examples, logic unit 660 may be configured to control information flow between units and direct services provided by API unit 665, input unit 670, output unit 675 in some embodiments described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 660 alone or in combination with API unit 665. Input unit 670 may be configured to obtain inputs for calculations described in the embodiments, and output unit 675 may be configured to provide outputs based on the calculations described in the embodiments.

[0038] The processor 610 may be configured to receive a request to control a humanoid from a remote user, as shown in FIG. 4. The processor 610 may also be configured to determine acceptance of the request by the host, as shown in FIG. 4. For a request determined to be accepted, the processor 610 may also be configured to verify receipt of a set of activation signals, as shown in FIG. 4. For a received set of activation signals, the processor 610 may also be configured to receive a humanoid mapping method selected by the remote user, as shown in FIG. 4, and control the humanoid based on the humanoid mapping method determined by the remote user. For a set of activation signals not received, the processor 610 may also be configured to terminate the request to control a humanoid, as shown in FIG. 4. The processor 610 may also be configured to broadcast an environment perceived from a first camera coupled to a humanoid in the remote user's VR equipment, as shown in FIG. 4.

[0039] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of predefined steps leading to a desired end state or result. In the illustrative examples, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0040] Unless otherwise specifically indicated, and as will be apparent from the discussion, throughout the description, discussion utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like will be recognized to include operations and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data that are similarly represented as physical quantities in the computer system's memory or registers or other information storage, transmission, or display device.

[0041] The embodiments may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. The computer-readable signal medium may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. The computer programs may include pure software implementations that include instructions to perform the operations of a desired implementation.

[0042] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove useful to construct more specialized apparatus to perform the desired method steps. Additionally, the examples are not described with reference to any particular programming language. It will be appreciated that various programming languages ​​may be used to implement the teachings of the implementations as described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0043] As known in the art, the operations described above may be implemented by hardware, software, or some combination of software and hardware. Various aspects of the embodiments may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform a method for implementing the examples of the present application. Furthermore, some embodiments of the present application may be implemented exclusively by hardware, while other embodiments may be implemented exclusively by software. Furthermore, the various functions described may be implemented in a single unit or may be spread across multiple components in various manners. When implemented by software, the method may be executed by a processor, such as a general-purpose computer, based on instructions stored on a machine-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted form.

[0044] That is, aspects of the disclosure may involve an innovative non-transitory computer readable medium storing instructions for implementing telexistence. The instructions may include receiving, by a processor, a request to control a humanoid from a remote user, determining, by the processor, acceptance of the request by the host, verifying, by the processor, receipt of a set of activation signals if the request is determined to be accepted, receiving a humanoid mapping method selected by the remote user if the set of activation signals is received and controlling the humanoid based on the humanoid mapping method determined by the remote user, and terminating, by the processor, the request to control the humanoid if the set of activation signals is not received.

[0045] Aspects of the present disclosure may also involve an innovative server system for implementing telexistence. The server system may include receiving, by a processor, a request to control a humanoid from a remote user, determining, by the processor, acceptance of the request by the host, verifying, by the processor, receipt of a set of activation signals if the request is determined to be accepted, receiving, by the processor, a humanoid mapping method selected by the remote user if the set of activation signals is received, and controlling the humanoid based on the humanoid mapping method determined by the remote user, and terminating, by the processor, the request to control the humanoid if the set of activation signals is not received.

[0046] Yet further, aspects of the present disclosure involve an innovative system for implementing telexistence, which may include means for receiving a request to control a humanoid from a remote user, means for determining acceptance of the request by the host, means for verifying receipt of a set of activation signals if the request is determined to be accepted, means for receiving a humanoid mapping method selected by the remote user if the set of activation signals is received and controlling the humanoid based on the humanoid mapping method determined by the remote user, and means for terminating the request to control the humanoid if the set of activation signals is not received.

[0047] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described examples may be used alone or in any combination. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the present application being indicated by the following claims. [Explanation of symbols]

[0048] 100:Humanoid 101: sensing system, 102: control system, 103: actuation system, 104: camera, 105: screen display, 200: remote configuration, 201: VR device, 202: camera, 203: display device, 300: remote viewer configuration, 301: display device, 302: VR device, 303: camera, 600: computing environment, 605: computing device, 610: processor, 615: memory, 620: internal storage, 625: IO interface, 630: bus, 635: user interface, 640: interface, 645: external storage, 650: network, 660: logic unit, 665: API unit, 670: input unit, 675: output unit, 695: inter-unit communication mechanism

Claims

1. receiving, by a processor, a request from a remote user to control the humanoid; determining, by the processor, acceptance of the request by a host; if it is determined that the request has been accepted, verifying, by the processor, receipt of a set of activation signals; If the set of activation signals is received, receiving a humanoid mapping method selected by the remote user and controlling the humanoid based on the humanoid mapping method determined by the remote user; terminating, by the processor, the request to control the humanoid if the set of activation signals is not received. How to implement telexistence.

2. 2. The method of claim 1, wherein the set of activation signals includes input signals associated with the remote user's virtual reality equipment, motion detection signals of the remote user, and a combined weight signal transmitted by the humanoid.

3. 3. The method of claim 2, further comprising: streaming the perceived environment from a first camera coupled to the humanoid in the virtual reality device of the remote user.

4. The method of claim 2 , wherein the motion detection signal is a feed signal obtained from a second camera pointed at the remote user.

5. The control of the humanoid based on the humanoid mapping method includes: Using the second camera to track the movements of the remote user; generating a motion prediction using a trained machine learning model in which the tracked motion of the remote user is used as an input; and performing control of the humanoid based on the generated motion prediction.

5. A method for implementing telexistence according to claim 4, comprising:

6. 2. The method of claim 1, wherein receiving the humanoid mapping method selected by the remote user includes receiving a selection from the remote user of either humanoid-to-user mapping or user-to-humanoid mapping.

7. When the humanoid-to-user mapping is selected as the humanoid mapping method, determining, by the processor, a combined weight load associated with the humanoid; if the combined weight load is determined to be significant, the processor generates a first safety alert for the remote user, detects a humanoid mapping method change request made by the remote user in response to the first safety alert, and grants control of the humanoid to the remote user if the humanoid mapping method change request is detected; if the combined weight load is not determined to be significant, granting control of the humanoid to the remote user; 7. A method for implementing telexistence according to claim 6.

8. If the humanoid mapping method change request is not detected, the processor further: generating a second safety alert for the remote user that confirms the humanoid mapping method; and granting control of the humanoid to the remote user if a humanoid mapping method is confirmed in response to the second safety alert.

8. The method for implementing telexistence according to claim 7, configured as follows:

9. 9. The method of claim 8, wherein if user-to-humanoid mapping is selected as the humanoid mapping method, the processor allows the remote user to control the humanoid.

10. the request to control the humanoid is received after a previous remote user's control of the humanoid in a previous telexistence session has ended; the orientation of the humanoid after the end of the previous telexistence session remains in a first state; if user-to-humanoid mapping is selected as the humanoid mapping method, mapping the body of the remote user to the orientation of the humanoid in the first state; if humanoid-to-user mapping is selected as the humanoid mapping method and the joint weight load is not determined to be significant, changing the orientation of the humanoid to a second state by mapping the humanoid onto the body of the remote user; if the humanoid-to-user mapping is selected as the humanoid mapping method and a request to change the humanoid mapping method is detected, mapping the body of the remote user to the orientation of the humanoid in the first state; if mapping from the humanoid to a user is selected as the humanoid mapping method and a request to change the humanoid mapping method is not detected, changing the orientation of the humanoid to the second state; 10. A method for implementing telexistence according to claim 9.

11. Humanoids and a processor in communication with the humanoid, receiving a request from a remote user to control the humanoid; determining acceptance of the request by the host; if it is determined that the request has been accepted, verifying receipt of a set of activation signals; If the set of activation signals is received, receiving a humanoid mapping method selected by the remote user; Controlling the humanoid based on the humanoid mapping method determined by the remote user; a processor configured to terminate the request to control the humanoid if the set of activation signals is not received; A system for implementing telexistence comprising:

12. The system of claim 11 , wherein the set of activation signals includes input signals associated with a virtual reality device of the remote user, a motion detection signal of the remote user, and a combined weight signal transmitted by the humanoid.

13. The system of claim 12 , wherein the processor is further configured to broadcast an environment perceived from a first camera coupled to the humanoid in the virtual reality device of the remote user.

14. The system of claim 12 , wherein the motion detection signal is a feed signal obtained from a second camera pointed at the remote user.

15. The processor, Using the second camera to track the movements of the remote user; generating a motion prediction using a trained machine learning model in which the tracked motion of the remote user is used as an input; Controlling the humanoid based on the generated motion prediction.

15. The system of claim 14, configured to control the humanoid based on the humanoid mapping method by:

16. 12. The system of claim 11, wherein the processor is configured to receive the humanoid mapping method by receiving a selection from the remote user of either humanoid-to-user mapping or user-to-humanoid mapping.

17. When the humanoid-to-user mapping is selected as the humanoid mapping method, the processor: determining a combined weight load associated with said humanoid; If the combined weight load is determined to be significant, generating a first safety alert for the remote user; detecting a request to change a humanoid mapping method made by the remote user in response to the first safety alert; and granting control of the humanoid to the remote user if the request to change the humanoid mapping method is detected; If the combined weight load is not determined to be significant, then granting control of the humanoid to the remote user. The system according to claim 16, configured to:

18. If the humanoid mapping method change request is not detected, the processor further: generating a second safety alert for the remote user that confirms the humanoid mapping method; and granting control of the humanoid to the remote user if a humanoid mapping method is confirmed in response to the second safety alert. The system of claim 17 configured to:

19. 20. The system of claim 18, wherein the processor is configured to grant control of the humanoid to the remote user if the user-to-humanoid mapping is selected as the humanoid mapping method.

20. the request to control the humanoid is received after a previous remote user's control of the humanoid in a previous telexistence session has ended; the orientation of the humanoid after the end of the previous telexistence session remains in a first state; if user-to-humanoid mapping is selected as the humanoid mapping method, mapping the body of the remote user to the orientation of the humanoid in the first state; if humanoid-to-user mapping is selected as the humanoid mapping method and the joint weight load is not determined to be significant, the processor is configured to change the orientation of the humanoid to a second state by mapping the humanoid onto the body of the remote user; if the humanoid-to-user mapping is selected as the humanoid mapping method and a request to change the humanoid mapping method is detected, mapping the body of the remote user to the orientation of the humanoid in the first state; if humanoid-to-user mapping is selected as the humanoid mapping method and a request to change the humanoid mapping method is not detected, the processor is configured to change the orientation of the humanoid to the second state.

20. The system of claim 19.

Citation Information

Patent Citations

  • Control device, network control method, and program

    JP2020170934A

  • Mobility Agent

    JP2022534797A

  • Remote control system, information processing method, and program

    WO2019225548A1

  • Robot control device, robot control method, and robot control system

    JP2020040155A