Remote control robot platform based on multi-modal sensory data

Abstract

The invention relates to remotely controlling a robotic platform based on multi-modal sensory data. The present disclosure provides methods and systems for remotely controlling robotic platforms in confined spaces or other similar spaces that are not suitable for direct human operation. The control is accomplished using multi-modal sensory data including at least two sensory response types, such as a binocular stereoscopic vision type, a binaural stereo audio type, a force-reflecting haptic manipulation type, a haptic type, and the like. Multi-modal sensory data is obtained by a robotic platform positioned in a confined space and transmitted to a remote control station external to the confined space, where the multi-modal sensory data is used to generate a representation of the confined space. The multi-modal sensory data may be used to provide a multi-sensory high-fidelity remote presentation for an operator of the remote control station and to allow the operator to provide more accurate user input. The input may be communicated to a robotic platform to perform various operations within the confined space.

Description

[0001] This application is a divisional application of Chinese patent application 201710383450.7, entitled "Remote Control Robot Platform Based on Multimodal Sensory Data", filed on May 26, 2017.

[0002] Cross-references to related applications

[0003] This application relates to two concurrently filed U.S. patent applications entitled “Multi-Tread Vehicles and Methods of Operating Thereof” (file number 16-0017-US-NP_BNGCP081US) and “Stereoscopic Camera and Associated Method of Varying a Scale of a Stereoscopic Image Pair” (file number 15-2607-US-NP), the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0004] The present invention provides methods and systems for controlling a robot platform, and more specifically, according to some embodiments, provides a robot platform that is remotely positioned in a confined space and controlled using remote and / or local control commands generated based on multimodal sensory data. Background Technology

[0005] Robotic platforms can be deployed in a variety of environments where direct human manipulation is not ideal. Teleoperated robotic systems, including robotic platforms, can be used in such environments to perform remote operations using input from a remotely located operator. However, the operator's perception of the operating environment is limited by the system's sensory fidelity level. For such systems to be effective, the operator must effectively represent the operating environment remotely using sufficient and realistic sensory feedback. Generally, higher sensory fidelity provided to the operator results in a greater sense of presence in the operating environment and more effective operational instructions from the operator. On the other hand, remote control can be very challenging when the lack of some sensory experience leads to limited situational awareness. In most conventional teleoperated robotic systems, the operator has limited information about the actual operating environment. The primary sensory feedback is visual. Even robotic platforms with sophisticated vision systems provide limited information to their operators. Humans naturally rely on multiple senses to understand their environment, not just vision. Restricting the operator to visual information constrains their ability to fully understand the environment and provide necessary instructions. Furthermore, typical teleoperated robotic systems suffer from so-called "monocular vision." Specifically, such systems include single-field-of-view cameras and displays that do not provide binocular stereoscopic vision. Deep cues are essential for understanding the environment and performing various operations within it, such as manipulation tasks, especially for delicate manipulation tasks. Every introduced variation impairs the operator's ability to work precisely and can lead to fatigue from prolonged use. Summary of the Invention

[0006] This invention provides a method and system for remotely controlling a robotic platform in a confined space or other similar space unsuitable for direct human operation. Control is achieved using multimodal sensory data, which includes at least two sensory response types, such as binocular stereo vision, binocular stereo audio, force-reflex tactile manipulation, and tactile sensation. The multimodal sensory data is acquired by the robotic platform positioned within the confined space and transmitted to a remote control station outside the confined space, whereby the multimodal sensory data is used to generate a representation of the confined space. The multimodal sensory data can be used to provide a high-fidelity multisensory remote representation for the operator at the remote control station and allows the operator to provide more accurate user input. This input can be transmitted to the robotic platform to perform various operations within the confined space.

[0007] In some embodiments, a method is provided for remotely controlling a robotic platform based on multimodal sensory data. The method may include locating the robotic platform, communicatively coupling the robotic platform to a remote control station, acquiring multimodal sensory data using two or more sensors on the robotic platform, transmitting at least a portion of the multimodal sensory data, and receiving remote control commands from the remote control station at the robotic platform. The multimodal sensory data may include at least two sensory response types. The at least two sensory response types may be selected from the group consisting of binocular stereo vision, binocular stereo audio, force-reflex tactile manipulation, and tactile types. Acquiring and transmitting the multimodal sensory data during the execution of this method may be repeated continuously. The method may involve increasing the remote control commands received from the remote control station. In some embodiments, the structure is an aircraft wing.

[0008] In some embodiments, the robotic platform is positioned within a confined space within the structure. When the robotic platform is positioned within a confined space, it can perform the transmission of at least a portion of multimodal sensory data.

[0009] In some embodiments, the method further includes generating local control commands at the robotic platform based on multimodal sensory data. The method may also include performing one or more operations within a confined space using the robotic platform based on the local control commands.

[0010] In some embodiments, the multimodal sensory data includes at least binocular stereo vision type, binocular stereo audio type, and force-reflex tactile manipulation type. In these embodiments, one or more operations may include drilling holes in components of the structure.

[0011] In some embodiments, multimodal sensory data may include at least binocular stereo vision type, binocular stereo audio type, force-reflex tactile manipulation type, and tactile type. In these embodiments, one or more operations include installing fasteners into the structure.

[0012] In some embodiments, the method further includes adding multimodal sensory data before transmitting at least a portion of the multimodal sensory data. The method may also include selecting at least a portion of the multimodal sensory data for transmission.

[0013] In some embodiments, the method further includes performing one or more operations within a confined space using the robot platform based on remote control commands received from a remote control station at the robot platform. For example, the one or more operations are selected from the group consisting of: changing the position of the robot platform within the confined space, drilling holes in components of the structure, installing fasteners into the structure, sealing the structure, spraying paint onto the structure, removing an object from the confined space, and inspecting the structure. The fidelity level of the multimodal sensory data may correspond to one or more operations. In some embodiments, the fidelity level of the multimodal sensory data changes over time. In some embodiments, one or more operations are also performed based on local control commands generated at the robot platform, such that the local control commands are combined with remote control commands to perform one or more operations.

[0014] In some embodiments, one or more operations include changing the position of the robotic platform within a confined space. In these embodiments, the multimodal sensory data may include at least binocular stereo vision type and stereo audio type.

[0015] In some embodiments, a local area network (LAN) is used to communicatively couple the robot platform to the remote control station. In the same or other embodiments, at least one wireless communication link is used to communicatively couple the robot platform to the remote control station. Alternatively, a global communication network is used to communicatively couple the robot platform to the remote control station.

[0016] The present invention also provides a method for remotely controlling a robot platform in a confined space based on multimodal sensory data. The method may include receiving multimodal sensory data from the robot platform located in the confined space, generating a representation of the multimodal sensory data by a remote control station, capturing user input at the remote control station, and transmitting remote control commands to the robot platform located in the confined space. The multimodal sensory data is received by a remote control station located outside the confined space and communicatively coupled to the robot platform. The multimodal sensory data includes at least two sensory response types selected from the group consisting of binocular stereo vision, binocular stereo audio, force-reflex tactile manipulation, and tactile types; and

[0017] In some embodiments, generating a representation of multimodal sensory data includes adding multimodal sensory data based on at least one of video spectrum, audio spectrum, spatial orientation, and proprioception. This representation can be a multisensory high-fidelity telerepresentation. In some embodiments, the user interface of the remote control station includes a binocular stereoscopic vision-type 3D display for presenting the multimodal sensory data. The user interface of the remote control station may include stereo speakers of a binocular stereoscopic audio type for presenting the multimodal sensory data.

[0018] In some embodiments, remote control commands represent one or more operations performed by a robotic platform within a confined space. The one or more operations may be selected from the group consisting of: changing the position of the robotic platform within the confined space, drilling holes in components of the structure, installing fasteners into the structure, sealing the structure, spraying paint onto the structure, removing objects from the confined space, and inspecting the structure.

[0019] In some embodiments, receiving multimodal sensory data and generating representations are performed continuously. Furthermore, remote control commands can be generated based on user input. A local area network (LAN) can be used to communicatively couple the robot platform to a remote control station. In the same or other embodiments, a global communication network is used to communicatively couple the robot platform to the remote control station.

[0020] The present invention also provides a robotic platform for operating in a confined space of a structure using multimodal sensory data. The robotic platform may include sensors for generating multimodal sensory data and a communication module for communicatively coupling to a remote control station located outside the confined space. The sensors may include at least two selected from the group consisting of binocular stereo vision sensors, binocular stereo audio sensors, force-reflective tactile manipulation sensors, and tactile sensors.

[0021] The present invention also provides a remote control station for controlling a robot platform using multimodal sensory data. The remote control station may include: a communication module for communicatively coupling to the robot platform and for receiving multimodal sensory data from the robot platform; and a user interface including an output device for generating a representation of the multimodal sensory data received from the robot platform. The multimodal sensory data includes at least two sensory response types. In some embodiments, the at least two sensory response types are selected from the group consisting of binocular stereo vision, binocular stereo audio, force-reflex tactile manipulation, and tactile types.

[0022] This invention also provides a method for remotely controlling a robotic platform in a confined space based on multimodal sensory data. The method may include acquiring multimodal sensory data using two or more sensors of the robotic platform, transmitting at least a portion of the multimodal sensory data to a remote control station, and generating a representation of the multimodal sensory data via the remote control station. The multimodal sensory data includes at least two sensory response types. Various other aspects of this method are presented above and elsewhere in this document.

[0023] These and other embodiments are further described below with reference to the accompanying drawings. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of an aircraft with a confined space according to some embodiments.

[0025] Figure 2 These are examples of collaborative robot systems including a robot platform and a remote control station, according to some embodiments.

[0026] Figure 3 This is a schematic diagram of a robot platform according to some embodiments.

[0027] Figure 4 This is a schematic diagram of a remote control station according to some embodiments.

[0028] Figure 5 This is a schematic diagram of multimodal sensory data according to some embodiments.

[0029] Figure 6 This is a process flowchart corresponding to a method for remotely controlling a robot platform in a confined space, according to some embodiments.

[0030] Figure 7 This is a process flowchart corresponding to a method for remotely controlling a robot platform in a confined space from the perspective of a remote control station, according to some embodiments.

[0031] Figure 8 This is a block diagram of an aircraft manufacturing and maintenance method that can utilize the methods and components described herein. Detailed Implementation

[0032] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the presented concepts. The presented concepts may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail so as not to unnecessarily obscure the described concepts. While some concepts will be described in conjunction with specific embodiments, it should be understood that these embodiments are not intended to be limiting. Rather, they are intended to cover alternatives, modifications, and equivalents that may be included within the spirit and scope of this disclosure as defined by the appended claims.

[0033] introduce

[0034] Adding real-time human control to robotic platforms presents new opportunities for robotics. On one hand, it helps overcome many of the challenges associated with fully automated systems. Furthermore, it allows for operations in environments inaccessible to humans and / or in situations not supported by fully automated systems.

[0035] Hybrid human-robot systems can safely leverage their respective strengths and achieve substantial synergies when presented with sufficient information about the operating environment to a human operator. For example, a key advantage of robotic platforms is their ability to deploy and operate in a variety of environments that are not easily accessible to humans, such as confined spaces, hazardous environments, etc. (For the purposes of this disclosure, a confined space is defined as an enclosed space defined by cavities and access points, wherein the cavity depth is at least five times the main dimension of the access point.) Meanwhile, humans are able to operate well in complex and unstructured environments using their sensory and cognitive abilities (which currently far exceed those of fully automated robotic systems). However, the complexity of these environments often involves access points in areas that are not easily accessible or suitable for humans. For example, the interior of an aircraft wing is a complex environment with many different components that may need to be assembled, repaired, and replaced. The size or (more specifically) thickness of the wing provides limited access to these components. It should be noted that access points also restrict access to the location, size, and other characteristics of the operating environment. Current wing designs offer a variety of access points designed for human operators. However, these access points may be undesirable from the perspectives of weight, performance, and other considerations, and should generally be smaller and less frequent if possible.

[0036] The effective operation of hybrid human-robot systems (also known as collaborative robot systems) depends on providing the operator with high-fidelity telerepresentation, enabling the operator to provide accurate user input. The level of fidelity depends on sensory data acquired through the robotic platform present in the operating environment, and particularly on the different sensory modalities of the data. In many cases, each individual sensory data type (e.g., vision) may not be sufficient for a human operator to have adequate perception of the operating environment. In most cases, humans rely on multiple senses to generate their environmental perception.

[0037] This invention provides apparatus, methods, and systems for controlling a robotic platform positioned in a confined space. This control is provided at least partially by a remote control station located outside the confined space. A degree of control may be provided by the robotic platform itself, which can be referred to as the automated portion of the overall control. Internal and / or external control is based on multimodal sensory data acquired by the robotic platform. This collaborative robotic approach removes human operators from the confined space and provides a safe and ergonomic working environment remotely. The method allows operations to be performed in environments where humans are not easily accessible. Furthermore, it opens doors to new types of operations that may not be performed directly by humans or assisted by fully automated robotic systems. It also opens doors to new structural configurations that no longer require an operating environment accommodating humans. For example, a larger ratio of chord (Y-dimensional) to depth (Z-dimensional) in the airfoil, a lighter structure that does not require human-sized access points, and other similar features could be used on aircraft. For the purposes of this disclosure, multimodal sensory data is defined as data generated by one or more sensors of the robotic platform positioned in the confined space and corresponding to at least two different types of human sensation (either directly or in an augmented form).

[0038] A degree of automation can be provided by optional autonomous agents that assist the operator in their control and / or can be responsible for selecting multimodal sensory data for transmission, and even modifying the multimodal sensory data (e.g., scaling, changing the sensory spectrum, etc.). These agents can be implemented on a robotic platform, a remote control station, or both. Specifically, the robotic platform can perform certain operations without any control instructions generated based on user input. Control instructions for these operations can be generated by one or more autonomous agents based on multimodal sensory data. Some examples of these operations may include navigating the robotic platform within a confined space based on the target location and proximity to various surrounding objects. Other examples may involve various operations with less complexity than, for example, operations performed based on user input.

[0039] In some embodiments, methods and systems create situational awareness for operators through immersive multisensory high-fidelity presentation or, more specifically, remote presentation. This type of situational awareness allows operators to generate user input more accurately and efficiently without actually being present in a constrained space where multimodal sensory data is available. In addition to increased efficiency, high fidelity allows for control of more complex operations. Similar to other instances of limitations in human sensory function, a limited sensory presence significantly constrains an operator's ability to perform tasks.

[0040] Collaborative robotic systems are designed to generate immersive multimodal sensory feedback (e.g., a combination of visual and auditory perception of the environment, and in some cases, extremity). Utilizing such feedback, operators working via remote control stations will have a more realistic sense of being in the environment and can employ the intuition and prudent practices that field workers would use while ensuring safety. Furthermore, the operator's intuition and practice (even for remotely positioned operators) can surpass some of the autonomy of current robotic platforms, making control commands generated based on user input (and understanding of the operating environment) truly invaluable. In some embodiments, the visual component of the multimodal sensory data can be implemented using a high-resolution (e.g., 1920 × 1280 pixels, with 24-bit luminance and color data per pixel at 60 Hz per eye) geometrically corrected binocular stereo remote-view sensor. The audio component can be full-range (e.g., 20 kHz bandwidth per ear) stereo audio linked to a microphone of the visual telepresence and reproduced to give the operator a natural auditory situational awareness.

[0041] Other sensor modalities may include scalable force-reflecting manipulators, scalable amplitude attitude platforms driven by one or more inertial measurement sensors or their end effectors on a remote platform, and remote tactile sensing utilizing fingertip arrays of pressure and temperature reproducing sensors and vibration reproduction. Other forms of sensory augmentation can be used, such as scaling (visual size and spectrum, force, and hearing). The type of sensory data can depend on the environment and the operation being performed. For example, a bulldozer operator could be presented with wide-angle, high-definition single-field-of-view video combined with full-range stereo audio, and an attitude reproduction platform with vibration reproduction. A racing driver could add wind speed and direction reproduction by blowing air into the operator's face for better situational awareness. Ambient temperature (e.g., air temperature) can also be used as a factor. A surgeon could be presented with scaled high-definition stereo vision and scaled force-reflecting manipulators. Finally, explosive ordnance disposal operators could add tactile sensing for better manipulation.

[0042] In some embodiments, when a confined environment is presented, for example, on a user interface at a remote control station, the methods and systems allow scaling of the representation of that confined environment. For example, an tool that performs an operation and is operated and controlled by an operator's hand may be substantially smaller than the operator's hand. Scaling can be used to represent the tool at a scale comparable to the size of a hand. It should be noted that different scaling can be used for different types of sensory data and even for different subsets of data of the same sensory data type. For example, visual representations may be scaled up, while force feedback may be scaled down (e.g., to avoid damaging the operator's hand). In other words, scaling is used to more effectively match the operator's perceptual and sensory abilities to a specific space and / or the task at hand.

[0043] For example, a one-to-one scaling system would give the manipulator the length of a human arm and allow it to move the same distance, which may not be suitable for environments requiring smaller or larger manipulators. Now, referring to visual scaling, a stereo camera can be positioned at a relative distance and location above the actuator, similar to the human eye relative to the hand. The stereo camera can have the same interpupillary distance as our eyes. For a two-to-one scaling increase effective in remote environments, the manipulator must be half the size of our arm, and the distance to the stereo camera and the height of the stereo camera must be half of the previous distance and height, where the interpupillary distance is half of our (human) interpupillary distance.

[0044] In some embodiments, methods and systems augment multimodal sensory data based on the operator's sensory capabilities. Augmentation can be performed on different types of sensory data (e.g., imaging, audio, force, temperature, etc.), and even one type can be converted into another (e.g., creating a visual (color) representation of a temperature map). This augmentation capability allows the use of sensory data that could otherwise be ignored if the operator were present in the actual operating environment. For example, an operator may not be able to see infrared radiation (e.g., a temperature indication) or hear sounds beyond the common 20 Hz to 20 kHz range (e.g., sounds outside this range can indicate a particular type of friction). Data collected for such ranges can be converted into a range recognizable by a human operator. Furthermore, one sensory type can be presented in the form of another sensory type. For example, the temperature distribution of a surface can be presented using different colors on a remote control station's user interface.

[0045] In some embodiments, the methods and systems provide precise physical interaction via tactile teleoperation. Specifically, at least one component of the multimodal sensory data can be based on tactile and proprioceptive sensations. Various types of sensors (e.g., force sensors, temperature sensors, etc.) can be used on a robotic platform to generate tactile sensory data. Furthermore, the user interface can include various tactile output devices to generate a representation of this data.

[0046] In general, complex tasks in unstructured environments are more difficult to characterize and represent than repetitive tasks in well-structured settings, where robotic advancements are currently prevalent and where full robotic automation may have become possible. Operation in unstructured environments still relies on human capabilities to understand the environment and provide at least some control instructions. However, operators require a sufficient representation of such unstructured environments, which is addressed by utilizing multimodal sensory data. Systems comprising a robotic platform and a remote control station can be termed collaborative robotic systems or hybrid robot-human systems, generating at least some control instructions for the robotic platform. This type of system utilizes the capabilities of each component of the system. Specifically, this type of system utilizes the capabilities of robotic platforms approaching those of environments unsuitable for humans, performs specific tasks within those environments, and acquires multimodal sensory data that, in some embodiments, can surpass human sensory capabilities. The system can support sensory and cognitive enhancements as described above.

[0047] Various sensors are positioned on the robot platform to generate multimodal sensory data. Each sensor may represent one end of a remote sensory channel. In some embodiments, the channel may include a monitor agent, which may be responsible for modifying and / or augmenting the data generated by the sensors and / or monitoring control commands from the operator. For example, the monitor agent may scale movement, limit acceleration, and / or apply soft limits to control commands. This scaling may be used to prevent collisions and other issues. In some embodiments, multimodal sensory data of different sensory types are analyzed simultaneously by the same monitor agent. Furthermore, the data may be presented simultaneously at a remote control station. Some examples of this data analysis include, but are not limited to, constructing a 3D map of the operating space (which may be visible to the operator at the user interface), identifying anomalous features (such as missing fasteners or surface defects), and combining / overlaying different sensory types on the user interface.

[0048] The collaborative robotic systems described herein may include multimodal remote sensing sensors for binocular stereo vision, binocular stereo hearing, and / or force-reflex tactile manipulation. The robotic platforms of these systems can be remotely controlled and deployed into confined / hazardous spaces. The robotic platforms may be particularly suited to the operating environment (e.g., space requirements, access, etc.) and the operations performed by these platforms. For example, access via relatively small loop tunnels may require a serpentine robotic platform, while on-orbit entry may require a free-flying robotic platform.

[0049] To provide a better understanding of the challenges associated with operations in confined spaces, reference will now be made to... Figure 1 An example describing a constrained space. Specifically, Figure 1This is a schematic diagram of an aircraft 100 according to some embodiments. The aircraft 100 includes a fuselage 150 having an interior 170. The aircraft 100 includes a wing 120 coupled to the fuselage 150. The aircraft 100 also includes an engine 130 coupled to the wing 120. In some embodiments, the aircraft 100 further includes a plurality of operating systems 140 and 160 (e.g., avionics), which are described below in conjunction with... Figure 8 Further description. Any of these aircraft components may have an operating environment that is difficult for humans to access, and at the same time too complex for fully autonomous robotic operation. For example, wing 120 may include various ribs and other structural components that restrict access to the interior of wing 120.

[0050] In general, robotic platforms can be deployed in restricted and / or high-risk areas where humans should not or cannot be sent. Robotic platforms can withstand a variety of risks associated with that environment and / or the operations performed within it. Risks can include accidental or unintentional actions such as falls, collisions, entanglement, or wedging. These actions are often the result of a lack of awareness of the environment by the human operator or various autonomous agents.

[0051] Examples of collaborative robot systems and their components

[0052] Figure 2 This is an example of a collaborative robot system 200 including a robot platform 230 and a remote control station 250, according to some embodiments. During operation of the collaborative robot system 200, the robot platform 230 is positioned within the confined space 210 of the structure 212. Although Figure 2 An example of structure 212 as an aircraft wing is shown, but those skilled in the art will understand that any other examples of structure 212 and confined space 210 are also within the scope. Some additional examples of structure 212 and confined space 210 of structure 212 include, but are not limited to, fuselage, rudder, horizontal stabilizer, flaps, slats, ailerons, keel, crown, or other restricted access areas of the aircraft. During operation of the collaborative robot system 200, remote control station 250 is positioned outside confined space 210, thereby allowing the operator to interact with remote control station 250. Although Figure 2 An access point is shown positioned at one end of structure 212, but those skilled in the art will understand that other examples of access points are also within this scope. For example, the access point may be provided within the wingtip, or more specifically, within the wing root of the wingtip. In another example, the access point may be in the crown or keel of the fuselage.

[0053] Robot platform 230 and remote control station 250 are communicatively coupled using, for example, a communication link 270. Communication link 270 can be a wired link, a wireless link, or various combinations of both. Various communication protocols and / or networks can be used to establish communication link 270. In some embodiments, communication link 270 can utilize a local area network (LAN), a global communication network (e.g., the Internet), etc. (Where does the power (hydraulic, electrical, etc.) run to 230? Is it possible to transmit power to 230 via an umbilical cable running in parallel with 270?). The choice of network and protocol depends on the proximity of robot platform 230 and remote control station 250, as well as other factors. Although not explicitly stated in the text... Figure 2 As shown, however, the collaborative robot system 200 may include power lines (e.g., hydraulic, electric, pneumatic, etc.) extending to the robot platform 230. Power can be supplied to the robot platform 230 from outside the confined space 210. In some embodiments, the power source may be inside the robot platform 230.

[0054] Multimodal sensory data is transmitted from robot platform 230 and remote control station 250 using communication link 270, thereby creating a high-fidelity immersive telepresence for the operator of remote control station 250. This type of telepresence provides situational awareness required for many operations. Simultaneously, establishing this telepresence may require high-fidelity capture and reproduction of sensory and sensorimotor data acquired by robot platform 230. The various types of data acquired by robot platform 230 are collectively referred to as multimodal sensory data 272. A schematic diagram of the various components of multimodal sensory data 272 is presented in… Figure 5 For example, a remote presentation to an operator may include a geometrically corrected binocular stereoscopic viewing system and high-fidelity stereo audio reproduction. In this example, multimodal sensory data 272 may include binocular stereoscopic vision type 273a and binocular stereo audio type 273b. In the same or other examples, a force-reflective manipulation sensor may be used to generate force-reflective tactile manipulation type 273c.

[0055] In some embodiments, the collaborative robot system 200 may include one or more additional remote control stations 250'. The additional remote control stations 250' may be communicatively coupled to the main remote control station 250 or directly coupled to the robot platform 230. Multiple remote control stations 250 and 250' may be used by different operators providing user input. Different remote control stations 250 and 250' may be located in substantially the same location (e.g., at the work site) or in different locations. For example, remote control station 250 may be a local station located substantially near the robot platform 230, while additional remote control stations 250' may be remote stations located at different locations. Control of the different remote control stations 250 and 250' may be performed by different parties. For example, the local remote control station 250 may be controlled by an aircraft operator (e.g., an airline), airport personnel, and / or maintenance service personnel, while the remote remote control station 250 may be controlled by the aircraft manufacturer or airline headquarters (e.g., with additional knowledge of structure 212).

[0056] The operator of the additional remote control station 250' can have more specific domain knowledge than, for example, the operator of remote control station 250, and can be able to support multiple different collaborative robot systems 200. This is particularly useful when detecting unforeseen situations that require additional specialized knowledge. By supporting collaborative access to various remote domain experts, unexpected situations can be resolved quickly without the time and cost of co-locating experts for consultation. For example, an airline could have a single trained expert operating collaborative robot system 200 instead of multiple collaborative robot systems 200 or at least multiple robot platforms 230. These multiple robot platforms 230 can be located at various facilities. The expert can be able to control each robot platform 230 when needed without co-locating with that robot platform 230.

[0057] Each component of the collaborative robot system 200 will now be described in more detail. Figure 3 This is a schematic diagram of a robot platform 230 according to some embodiments. The robot platform 230 includes various sensors 510 for acquiring multimodal sensory data. Some examples of sensors 510 include, but are not limited to, a binocular stereo vision sensor 512, a binocular stereo audio sensor 514, a force-reflective tactile manipulation sensor 516, a tactile sensor 518, and a temperature sensor 517. The multimodal sensory data is a combination of the outputs of two or more of these sensors.

[0058] The selection of sensors 510 on robot platform 230 can depend on specific aspects of the multimodal sensory data. The sensory experience generated at remote control station 250 based on the multimodal sensory data obtained by sensors 510 on robot platform 230 can be selected for each specific operation, as shown in Table 1.

[0059] Table 1

[0060]

[0061] Binocular stereo vision is most useful for any manipulating task (such as placing a fastening tool on a fastener or grasping a fallen fastener) and inspection task (such as confirming the continuous outline of a sealant bead or distinguishing a mark that does not need repair from a scratch that does). Binocular stereo audio can be useful for situational awareness. For example, binocular stereo vision can provide perception of the operating environment based on sound generated by a robotic platform and reflected from the inner surface of a confined space. It is also useful for listening to tools (such as drill bits used for problems such as dulling or breakage). For example, a collaborative robotic system can have software that automatically monitors the acoustic portion of multimodal sensory data to detect operational defects (e.g., suboptimal drilling can be characterized by different sounds). Force-reflective tactile proprioception is useful for placing fasteners in holes or applying pressure and resistance to a cleaning pad during wiping. In some embodiments, a collaborative robotic system can generate feedback to the operator and / or one or more automated agents to monitor the application of forces in order to more closely follow the operation (e.g., detecting suboptimal drilling conditions during drilling). Tactile sensing is useful for fine manipulation of tools or parts. Finally, vestibular spatial orientation is useful for providing intuitive perception of the orientation of remote vehicles or end effectors by offering angles and accelerations. It is also useful for detecting vibrations caused by workpiece movement or scratching.

[0062] Visual data type 273a, or more specifically, binocular stereo vision data, can be part of the overall multimodal sensory data 272. Visual data can be acquired using an optional stereo vision sensor 512, or more specifically, a geometrically corrected binocular stereo camera and observation system.

[0063] The vision sensor 512 can achieve geometrically corrected image capture by utilizing two coplanar camera sensors 513. The camera sensors 513 can be modified to offset the center of each sensor from the lens optical axis to shift the field of view, thereby allowing the visual area of ​​the field of view to be consistent. This particular arrangement of the camera sensors 513 produces geometrically corrected images that are unattainable using conventional stereo cameras.

[0064] It is insufficient to consider only the camera sensor 513 in the geometrically corrected telepresence observation system. To reproduce a scene as if the operator were looking at it with uninsulated eyes, the user interface 610 of the remote control station 250 may include a specific output device 612, such as a display 613a, that also conforms to the equivalent geometry. For example, when visual data is presented on the display 613a, the display 613a is naturally perceived as a window through which the operator is looking. By strictly adhering to the equivalent geometry of the direct view through the window used for the binocular stereo vision sensor 512 and the view of the virtual image through the screen of the display 613, the target scene can be accurately reproduced.

[0065] Specifically, the interpupillary distance of the operator's eyes can be a fixed measurement. When the operator positions his or her eyes such that a line drawn through both pupils is parallel to the window, and the cyclopean (the point between the two pupils) is positioned perpendicular to the plane of the window and centered on the window aperture, the width of the window constrains the viewing angle of each eye and defines the overlapping area. The effective choice of window aperture is limited by the physical width of the display screen. The geometric constraints of the system are completed by merging the distance of the viewer's eyes from the display screen.

[0066] Referring now to anthropomorphic hearing, many subtle depth and manipulation cues are subconsciously processed through human stereoscopic hearing. Simultaneously, adding this sensory modality to the collaborative robot system 200 is relatively simple, while providing a wealth of situational awareness of the remotely perceived environment. Binaural auditory data at a normal human scale can be provided by a stereo audio sensor 514 or more specifically by anthropomorphically calibrated stereo microphone 515. The stereo audio sensor 514 is, for example, as... Figure 3 This is part of the robot platform 230 shown. For scaled stereo hearing (e.g., to complement the scaled stereo vision identified above), the stereo audio sensor 514 can be a high-fidelity miniature microphone. While many of the same binaural localization cues (e.g., intensity, timbre, spectral quality, reflections in confined spaces) can be maintained, timing and phase cues in certain frequency bands can be reduced or altered, for example, if the inter-wall distance is changed.

[0067] With reference to force-reflex tactile telemanipulation, humans are highly capable of navigating (e.g., walking through a dark room) using touch and gentle bumping into objects without vision. Force-reflex tactile manipulation type 273c can be low-bandwidth type multimodal sensory data 272. Force feedback actuators and posture proprioception are added to facilitate sensorimotor control in order to interact with the remote environment. For example, a collaborative robotic system 200 can utilize a robotic hand and arm (which can be part of sensor 510 and includes both output device 612 and input device 614) controlled by a force-reflex exoskeleton (which can be part of user interface 610 and includes both output device 612 and input device 614). This approach allows the operator to perform a wide variety of actions naturally. In some embodiments, the remote control station 250 includes a 4-axis force feedback arm, a bi-finger force feedback hand, and a force-reflex exoskeleton for the fingers and arm. The reproduction of total force on the hand allows for proprioception or kinesthetic sense (which is the self-sensory perception of the position of limbs and other parts of the body). This provides important additional cues to immersive visual remote perception and the overall addition of multimodal sensory data 272.

[0068] Referring now to vestibular spatial orientation, the posture (orientation) of robot platform 230 or its operating tool (e.g., end effector) can be relayed to remote control station 250 as part of multimodal sensory data 272. This posture can be reproduced by adjusting the posture of the platform or operator's seat to achieve vestibular spatial orientation feedback. This feedback can be performed at a relatively low frequency compared to other sensory types. Furthermore, this feedback can be scaled and / or limited for safety and other reasons (e.g., to prevent the operator from tilting beyond a critical point (e.g., effectively causing the operator to fall) while providing this feedback). An inertial measurement unit will be incorporated into the remote robotic system and relayed to three actuators of the support platform that will drive the remote administrator.

[0069] The robot platform 230 also includes a communication module 540 for communicatively coupling to a remote control station 250 located outside the confined space 210. Some examples of the communication module 540 include a modem (wired or wireless). In some embodiments, the communication module 540 is a wireless communication module.

[0070] In some embodiments, the robot platform 230 further includes an operating tool 520 for performing one or more operations within a confined space 210. Some examples of the operating tool 520 include, but are not limited to, a drilling machine, a rivet gun, a sealant applicator, and an inspection device.

[0071] In some embodiments, the robot platform 230 further includes a drive mechanism 530 for changing the position of the robot platform 230 within the confined space 210. One example of the drive mechanism 530 is a set of pedals coupled to a motor. However, other examples are also included. While the robot platform 230 is in Figure 2 The vehicle is depicted as being stepped on, but any type of robotic platform 230 capable of generating multimodal sensory data falls within this scope.

[0072] In some embodiments, the robot platform 230 and / or remote control station 250 may include one or more optional auxiliary agents ( Figure 3 Box 550 and Figure 4 The auxiliary agent (box 650) assists a human operator in performing various control operations on the collaborative robot system 200. Specifically, the auxiliary agent can utilize multimodal sensory data obtained by the robot platform 230 to provide a degree of control to the robot platform 230, modify the multimodal sensory data before generating a representation of the data for the operator, and / or modify control instructions generated based on user input. This provides a degree of automation. For example, the auxiliary agent can autonomously monitor, interpret, instruct, automate, and restrict the operation of the robot platform 230. In some embodiments, one or more task domains of the collaborative robot system 200 are analyzed and defined, allowing for its modular development, testing, and integration using the auxiliary agent. For example, the auxiliary agent can perform navigation functions, task-specific planning, and monitoring. The collaborative robot system 200 supports fall-forward / backward cooperation between one or more of its autonomous agents and user input.

[0073] Figure 4 This is a schematic diagram of a remote control station 250 according to some embodiments. The remote control station 250 includes one or more representations for generating multimodal sensory data and / or a user interface 610 for capturing user input. Specifically, the user interface 610 may include one or more output devices 612, some examples of which include, but are not limited to, displays (e.g., 3-D displays) and speakers (e.g., a set of stereo speakers). Furthermore, the user interface 610 may include one or more input devices 614.

[0074] The remote control station 250 also includes a communication module 640 for communicatively coupling to the robot platform 230, which is located within the confined space 210. The communication module 640 may be of the same type as the communication module of the platform 230. In some embodiments, the remote control station 250 further includes a processor 630 for generating control instructions for the robot platform 230 and a memory 635 for storing these instructions and multimodal sensory data 272.

[0075] As described above, the remote control station 250 may also include one or more optional auxiliary agents 650. Operation of the user interface 610 can be integrated with the operation of the auxiliary agents 650, allowing the multimodal sensory data to be modified before it is presented on the user interface 610. In some embodiments, user input captured by the user interface 610 can be modified by the auxiliary agent 650 before control instructions for the robot platform 230 are generated.

[0076] In some embodiments, advanced human supervision of autonomous actions is supported by intelligent auxiliary agents. This approach incorporates greater autonomy into a variety of tasks, such as safe path planning and navigation, automated task-specific operations, or system "health" monitoring.

[0077] In general, the remote control station 250 can be the center for planning, control, and collaboration of the entire collaborative robot system 200. The remote control station 250 can be involved in the mobility, manipulation, remote sensing, autonomous agent tasks, and other operations of the collaborative robot system 200. The remote control station 250 can provide a portal to facilitate collaboration among remote experts (e.g., including multiple experts and / or auxiliary agents).

[0078] The remote control station 250 supports direct human operation, or more specifically, remote operation by providing situational awareness presented in immersive multi-sensory high fidelity. Furthermore, the remote control station 250 can provide precise physical interaction via haptic remote operation. Increased human operation can be supported by autonomous agents, for example, to monitor safety and assist workers. Scaling various aspects of multimodal sensory data (e.g., scaling visual data) provides a better match between the actual environment and the operator's perception and sensation. Additionally, the remote control station 250 can provide an enhanced spectrum of visual, auditory, spatial orientation, and proprioceptive senses.

[0079] Examples of operating collaborative robot systems and their components

[0080] Figure 6 This is a process flowchart according to some embodiments corresponding to a method 300 for remotely controlling a robot platform 230 in a confined space 210. Control is performed based on multimodal sensory data. Specifically, method 300 refers to operations performed by the robot platform 230. Reference is made below. Figure 7 The description refers to operations performed at or by remote control station 250. Those skilled in the art will understand that the two sets of operations are part of the same operational scheme of the collaborative robot system 200, even if they can be performed by different parties, such as one party controlling the operation of robot platform 230 and the other party controlling the operation of remote control station 250.

[0081] Method 300 may begin during operation 310 by positioning the robot platform 230 within the confined space 210 of structure 212. Structure 212 may be an aircraft wing or any other structure, such as one that may not be suitable for human operation. This positioning operation may involve propelling (e.g., driving) the robot platform 230 into and within the confined space 210 based on control commands generated (e.g., autonomous or semi-autonomous motion) at the robot platform 230 and / or generated at a remote control station 250 and transmitted to the robot platform 230. Specifically, the robot platform 230 may include, as referenced above... Figure 3 The drive mechanism 530 is further described, and this drive mechanism 530 can be used to position the robot platform 230 within the confined space 210. Alternatively, the robot platform 230 may not have any drive mechanism, and it may be manually positioned within the confined space 210.

[0082] It should be noted that when the robot platform 230 is positioned within the confined space 210, the robot platform 230 is communicatively coupled to a remote control station 250 located outside the confined space 210. In some embodiments, method 300 may involve operations such as establishing a communication coupling between the robot platform 230 and the remote control station 250, such as... Figure 6 Operation 312 is shown.

[0083] Once the robotic platform 230 is positioned within the confined space 210, method 300 can continue to acquire multimodal sensory data 272 during operation 314. Multimodal sensory data 272 can be acquired using two or more sensors 510 of the robotic platform 230. Multimodal sensory data 272 can include at least two different types 273 of sensory responses, such as binocular stereo vision type 273a, binocular stereo audio type 273b, force-reflex tactile manipulation type 273c, and tactile type 273b. Various aspects of multimodal sensory data 272 have been described above.

[0084] In some embodiments, method 300 involves augmenting multimodal sensory data 272 during optional operation 316. Typically, multimodal sensory data 272 may be augmented before it is transmitted to remote control station 250 (e.g., at robot platform 230) or after transmission (e.g., at remote control station 250). In either case, the augmentation of multimodal sensory data 272 may be performed by an autonomous agent. Specifically, the agent may autonomously monitor, interpret, instruct, automate, and limit multimodal sensory data 272.

[0085] For example, a visual augmentation agent can address the autonomous detection of visual features of interest. This agent can use, for example, a 3D visual overlay presented to the user interface to identify these features to the operator. The overlay can be aligned with an actual image of the environment. The operator can be able to turn off the visual augmentation agent to reduce distraction. The feature set can be selected from task-specific needs and can include the automated detection of missing parts (such as fasteners, coating defects, and items that shouldn't be there (foreign debris)), such as fallen fasteners or tools, or excess coating material. Different types of inspections are possible within this scope.

[0086] Another example of a vision-enhanced agent is the use of sensors to address localization within a confined workspace. For instance, the agent can be used to construct a 3D map of the workspace based on one or more components of multimodal sensory data, such as visual and / or tactile components. A separate virtual display can be presented to the operator. This display can show the mapped space and current position of the robotic platform 230 within the space. This reference map can provide a higher level of situational awareness without immersion. This example of a vision-enhanced agent can also include controls such as viewpoint adjustment.

[0087] Another example is a physical augmentation agent that can provide constraints on force reflections, forces, and selective scaling of motion based on task and operator requirements. The same or another physical augmentation agent can utilize the aforementioned positioning data. For example, based on the mapped workspace and the position and orientation of the deployed robotic platform 230, the intelligent assistance agent can determine safe working areas and restricted areas (e.g., to prevent unwanted collisions or damage). These areas can be updated in real time as the multimodal sensory data 272 is acquired.

[0088] One additional example is scaling. Scaling of one or more types of multimodal sensory data 272 can be used to present data 272 in a format more naturally understood by the operator, such as being more in line with the operator's scale. This type of scaling can be referred to as human operator scaling. In some embodiments, scaling can be performed using one or more of the agents described above. Scaling is a powerful extension of the perceptual capabilities of visual and / or other components. Modifying the effective size of multimodal sensory data 272 of visual type 273a can be done without involving a change in the magnification of camera 513, as this introduces depth distortion along the optical axis. The visible geometry of scaling can be achieved by changing the interpupillary distance of the camera lens.

[0089] Method 300 can continue to transmit multimodal sensory data during operation 320. The data is transmitted from a robotic platform 230 located in confined space 210 to a remote control station 250 located outside confined space 210.

[0090] Then, method 300 continues to receive remote control commands from remote control station 250 during operation 330. These remote control commands are received by robot platform 230 using communication link 270 and generated by remote control station 250. The generation of these remote control commands will be referenced below. Figure 7 Further description. In short, these remote control commands can be generated based on user input 254 and / or by various auxiliary agents 650 available at the remote control station 250. The remote control commands should be distinguished from the local control commands generated by the robot platform 230.

[0091] In some embodiments, method 300 involves generating local control commands at robot platform 230 during optional operation 334. The local control commands may be generated based on multimodal sensory data.

[0092] In some embodiments, method 300 further includes performing one or more operations within a confined space using a robotic platform during optional operation 350. Operation 350 may be performed based at least on remote control instructions received from remote control station 250 at robotic platform 230. In some embodiments, local control instructions may also be used for operation 350. Some examples of operation 350 include, but are not limited to: changing the position of robotic platform 230 within confined space 210 (box 350a), drilling a hole in component 214 of structure 212 (box 350b), installing fasteners into structure 216 (box 350c), sealing structure 212 (box 350d), spraying paint onto structure 212 (box 350e), removing an object from confined space 210 (box 350f), and inspecting structure 212 (box 350g). Those skilled in the art will understand that various other examples of operation 350 are also within this scope.

[0093] For example, the operation could be changing the position of the robotic platform within a confined space. In this example, the multimodal sensory data could include at least binocular stereo vision and stereo audio.

[0094] In another example, the manipulation could include drilling holes in components of the structure. Multimodal sensory data could include at least binocular stereoscopic vision, stereoscopic audio, and force-reflex tactile manipulation.

[0095] In yet another example, the operation could include installing fasteners into the structure. Multimodal sensory data could include at least binocular stereoscopic vision type, stereoscopic audio type, force-reflex tactile manipulation type, and tactile type.

[0096] The fidelity level of multimodal sensory data can correspond to one or more operations. Some operations may require a higher fidelity level than others. Furthermore, the fidelity level of multimodal sensory data can change over time.

[0097] Figure 7 This is a process flowchart according to some embodiments of a method 400 for remotely controlling a robot platform 230 in a confined space 210 from the perspective of a remote control station 250. This control is performed based on multimodal sensory data. The operation of method 400 is performed by the remote control station 250. References above. Figure 6 Describes the operations performed at or by the robot platform 230.

[0098] Method 400 may begin by receiving multimodal sensory data from robot platform 230 during operation 420. During this receiving operation, robot platform 230 is positioned within confined space 210. The multimodal sensory data is received by remote control station 250 located outside confined space 210. Furthermore, remote control station 250 is communicatively coupled to robot platform 230. As described above, the multimodal sensory data may include at least two of the following sensory response types: binocular stereo vision type, binocular stereo audio type, force-reflex tactile manipulation type, and tactile type.

[0099] Method 400 may continue to generate a representation of the multimodal sensory data at a remote control station during operation 430. In some embodiments, the representation generation operation includes augmenting the multimodal sensory data based on at least one of video spectrum, audio spectrum, spatial orientation, and proprioception. The representation may be a multisensory high-fidelity telerepresentation.

[0100] In some embodiments, a representation is generated on a user interface 610 of the remote control station 250. The user interface 610, or more specifically, the output device 612 of the user interface 610, may include a 3D display 613a that generates 3D images based on multimodal sensory data. In the same or other embodiments, the user interface 610 includes a stereo speaker 613b that generates stereo sound on a binaural stereo audio type based on the multimodal sensory data.

[0101] Method 400 may continue to capture user input at the remote control station during operation 440. Remote control instructions may be generated based on user input. In some embodiments, at least some of the remote control instructions are generated by the remote control station 250 without user input.

[0102] Method 400 may continue to transmit remote control commands to robot platform 230 during operation 460. During this operation, robot platform 230 is positioned within confined space 210. The remote control commands may represent one or more operations performed by robot platform 230 within confined space 210. Some examples of these operations are presented above.

[0103] In some embodiments, data receiving operation 420 and representation generation operation 430 are performed continuously.

[0104] Examples of aircraft and methods for manufacturing and operating aircraft

[0105] In such Figure 8 The aircraft manufacturing and maintenance method 1200 shown, and as such Figure 1 Examples of this disclosure can be described within the context of the illustrated aircraft 100. During pre-production, exemplary method 1200 may include the specification and design of aircraft 100 (box 1204) and material procurement (box 1206). During production, the manufacturing of parts and sub-assemblies of aircraft 100 (box 1208) and the inspection system integration (box 1210) may be performed. The described methods and components may involve remotely controlling a robotic platform based on multimodal sensory data as described above, and the described methods and components may be used in any of the specification and design of aircraft 100 (box 1204), material procurement (box 1206), manufacturing of parts and sub-assemblies (box 1208), and / or the inspection system integration of aircraft 100 (box 1210).

[0106] Subsequently, aircraft 100 may undergo certification and delivery (box 1212) to enter service (box 1214). While in service, aircraft 100 may be scheduled for routine maintenance and repair (box 1216). Routine maintenance and repair may include modification, reconfiguration, refurbishment, etc., of one or more inspection systems of aircraft 100. The described methods and components may relate to a remotely controlled robotic platform based on the multimodal sensory data described above. This method may be used in any of the certification and delivery (box 1212), service (box 1214), and / or routine maintenance and repair (box 1216).

[0107] Each step in the process of exemplary method 1200 can be performed or implemented by an inspection system integrator, a third party, and / or an operator (e.g., a customer). For the purposes described, the inspection system integrator can include, but is not limited to, any number of aircraft manufacturers and primary inspection system subcontractors; the third party can include, but is not limited to, any number of vendors, subcontractors, and suppliers; and the operator can be an airline, leasing company, military entity, service organization, etc.

[0108] like Figure 1As shown, an aircraft 100 produced by the exemplary method 1200 may include a fuselage 150 having an interior 170. As previously described, the aircraft 100 further includes a wing 120 coupled to the fuselage 150, wherein an engine 130 is coupled to the wing 120. The fuselage 150 further includes multiple advanced inspection systems, such as an electrical inspection system 140 and an environmental inspection system 160. Any number of other inspection systems may be included. Although an aerospace example is shown, the principles disclosed herein can be applied to other industries, such as the automotive industry. Therefore, in addition to aircraft 100, the principles disclosed herein can be applied to other vehicles, such as land vehicles, sea vehicles, space vehicles, etc.

[0109] During any one or more phases of the manufacturing and maintenance method (exemplary method 1200), the apparatus(s) and methods(s) shown or described herein may be employed. For example, a component or sub-component corresponding to the component and sub-component manufacturing (box 1208) may be made or manufactured in a manner similar to that of a component or sub-component produced when the aircraft 100 is in service (box 1214). Furthermore, one or more examples of the apparatus(s), methods(s), or combinations thereof may be utilized during the production phases (box 1208) and (box 1210), for example, by substantially accelerating the assembly of the aircraft 100 or reducing the cost of the aircraft. Similarly, one or more examples of the apparatus or methods implemented therein, or combinations thereof, may be utilized, for example, but not limited to, when the aircraft 100 is in service (box 1214) and / or during maintenance and repair (box 1216).

[0110] in conclusion

[0111] The various examples of the apparatus(s) and methods(s) disclosed herein include a variety of components, features, and functions. It should be understood that the various examples of the apparatus(s) and methods(s) disclosed herein may include any components, features, and functions of any other example of the apparatus(s) and methods(s) disclosed herein in any combination, and all such possibilities are intended to fall within the spirit and scope of this disclosure.

[0112] In view of the benefits of this disclosure with respect to the teachings presented in the foregoing description and related figures, those skilled in the art will conceive of many modifications to the examples set forth herein.

[0113] Therefore, in summary, according to the first aspect of the invention, the following is provided:

[0114] A1. A method (300) for remotely controlling a robot platform (230) based on multimodal sensory data (272), the method (300) comprising:

[0115] Positioning (310) robot platform (230),

[0116] The robot platform (230) is communicatively coupled to the remote control station (250);

[0117] Multimodal sensory data (272) is obtained (314) using two or more sensors (510) of the robotic platform (230).

[0118] Multimodal sensory data (272) includes at least two types of sensory responses;

[0119] Transmit (320) at least a portion of multimodal sensory data (272); and

[0120] The robot platform (230) receives remote control commands (330) from the remote control station (250).

[0121] A2. The method (300) according to paragraph A1 also provides that the robot platform (230) is positioned in the confined space (210) of the structure (212).

[0122] A3. The method (300) according to paragraph A2 also provides that, when the robot platform (230) is positioned in a confined space (210), at least a portion of the multimodal sensory data (272) is transmitted (320).

[0123] A4. The method (300) according to paragraph A1 also provides that at least two sensory response types are selected from the group comprising binocular stereoscopic type (273a), binocular stereo audio type (273b), force-reflex tactile manipulation type (273c) and tactile type (273d).

[0124] A5. The method (300) according to paragraph A1 also provides that it further includes generating (334) local control commands at the robot platform (230) based on multimodal sensory data (272).

[0125] A6. The method (300) according to paragraph A5 also provides that it further includes using a robot platform (230) to perform (350) one or more operations within a confined space (210) based on local control instructions.

[0126] A7. The method (300) according to paragraph A6 also provides that the multimodal sensory data (272) includes at least a binocular stereo vision type (273a), a binocular stereo audio type (273b) and a force-reflex tactile manipulation type (273c).

[0127] A8. The method (300) according to paragraph A7 also provides that one or more operations (350) include drilling (350b) into the components (214) of the structure (212).

[0128] A9. The method (300) according to paragraph A6 also provides that the multimodal sensory data (272) includes at least a binocular stereo vision type (273a), a binocular stereo audio type (273b), a force-reflex tactile manipulation type (273c) and a tactile type (273d).

[0129] A10. The method (300) according to paragraph A9 also provides that one or more operations (350) include mounting (350c) a fastener to the structure (212).

[0130] A11. The method (300) according to paragraph A1 also provides that the acquisition (314) of multimodal sensory data (272) and the transmission of multimodal sensory data (272) are repeated continuously.

[0131] A12. The method (300) according to paragraph A1 also provides that it further includes adding (316) the multimodal sensory data (272) before transmitting (320) at least a portion of the multimodal sensory data (272).

[0132] A13. The method (300) according to paragraph A1 also provides that it further includes selecting (318) at least a portion of the multimodal sensory data (272) for transmission.

[0133] A14. The method (300) according to paragraph A1 also provides that it further includes using the robot platform (230) to perform (350) one or more operations within a confined space (210) based on remote control instructions received at the robot platform (230) from the remote control station (250).

[0134] A15. As provided in the method (300) of paragraph A14, one or more operations are selected from the group comprising:

[0135] Change the position (350a) of the robot platform (230) within the confined space (210).

[0136] Drill holes (350b) in component (214) of structure (212).

[0137] Install the fastener (350c) onto the structure (212).

[0138] The structure (212) is sealed (350d).

[0139] Spray (350e) onto structure (212),

[0140] Remove the object (350f) from the confined space (210), and

[0141] Examine the structure (212) of (350g).

[0142] A16. The method (300) according to paragraph A14 also provides that the fidelity level of the multimodal sensory data (272) corresponds to one or more operations.

[0143] A17. The method (300) in paragraph A14 also provides that the fidelity level of the multimodal sensory data (272) changes over time.

[0144] A18. The method (300) according to paragraph A14 also provides that one or more operations are performed based on local control instructions generated at the robot platform (230), such that the local control instructions are combined with remote control instructions to perform one or more operations.

[0145] A19. The method (300) according to paragraph A1 also provides that one or more operations include changing (350a) the position of the robot platform (230) within a confined space (210), and wherein the multimodal sensory data (272) includes at least a binocular stereo vision type (273a) and a stereo audio type (273b).

[0146] A20. The method (300) according to paragraph A1 also provides that it further includes adding (336) remote control commands received from the remote control station (250).

[0147] A21. According to the method (300) in paragraph A1, it is also provided that the structure (212) is an aircraft wing.

[0148] A22. The method (300) according to paragraph A1 also provides that the robot platform (230) is communicatively coupled to the remote control station (250) using a local area network.

[0149] A23. The method (300) according to paragraph A1 also provides that the robot platform (230) is communicatively coupled to the remote control station (250) using at least one wireless communication link.

[0150] A24. The method (300) according to paragraph A1 also provides that the robot platform (230) is communicatively coupled to a remote control station (250) using a global communication network.

[0151] According to another aspect of the invention, the following is provided:

[0152] B1. A method (400) for remotely controlling a robotic platform (230) in a confined space (210) of a structure (212) based on multimodal sensory data (272), the method (400) comprising:

[0153] Multimodal sensory data (272) is received (420) from a robotic platform (230) positioned in a confined space (210).

[0154] Multimodal sensory data (272) is received by a remote control station (250) located outside the confined space (210) and communicatively coupled to the robot platform (230).

[0155] Multimodal sensory data (272) includes at least two sensory response types selected from the group comprising binocular stereo vision type (273a), binocular stereo audio type (273b), force-reflex tactile manipulation type (273c), and tactile type (273d); and

[0156] Representation of multimodal sensory data (272) generated (430) by remote control station (250);

[0157] Capture (440) user input at the remote control station (250); and

[0158] Remote control commands are transmitted (460) to the robot platform (230) located in a confined space (210).

[0159] B2. The method (400) according to paragraph B1 also provides that the representation of generating (430) multimodal sensory data (272) includes augmenting the multimodal sensory data (272) based on at least one of video spectrum, audio spectrum, spatial orientation and proprioception.

[0160] B3. According to the method (400) in paragraph B1, it is also provided that it represents multi-sensory high-fidelity telepresence.

[0161] B4. The method (400) according to paragraph B1 also provides that the user interface (610) of the remote control station (250) includes a 3D display (613a) for presenting the binocular stereoscopic vision type (273a) in the multimodal sensory data (272).

[0162] B5. The method (400) according to paragraph B1 also provides that the user interface (610) of the remote control station (250) includes a stereo speaker (613b) for presenting the binaural stereo audio type (273b) in the multimodal sensory data (272).

[0163] B6. The method (400) according to paragraph B1 also provides that the remote control instructions represent one or more operations performed by the robot platform (230) within a confined space (210).

[0164] B7. As provided in the method (400) of paragraph B6, one or more of the operations are selected from the group comprising:

[0165] Change the position (350a) of the robot platform (230) within the confined space (210).

[0166] Drill holes (350a) in component (214) of structure (212),

[0167] Install the fastener (350a) onto the structure (212).

[0168] The structure (212) is sealed (350a).

[0169] Spray (350a) onto structure (212),

[0170] Remove (350a) object from confined space (210), and

[0171] Check structure (212) of (350a).

[0172] B8. The method (400) according to paragraph B1 also provides that at least (420) multimodal sensory data (272) is received and (430) representations are generated continuously.

[0173] B9. The method (400) according to paragraph B1 also provides that remote control commands are generated based on user input.

[0174] B10. The method (400) according to paragraph B1 also provides that the robot platform (230) is communicatively coupled to the remote control station (250) using a local area network.

[0175] B11. The method (400) according to paragraph B1 also provides that the robot platform (230) is communicatively coupled to a remote control station (250) using a global communication network.

[0176] According to another aspect of the invention, the following is provided:

[0177] C1. A robotic platform (230) for operating in a confined space (210) of a structure (212) using multimodal sensory data (272), the robotic platform (230) comprising:

[0178] Sensor (510), which is used to generate multimodal sensory data (272); and

[0179] A communication module (540) is used for communicationally coupled to a remote control station (250) located outside a confined space (210).

[0180] C2. The robot platform (230) according to paragraph C1, wherein the sensor (510) includes at least two selected from the group comprising a binocular stereo vision sensor (512), a binocular stereo audio sensor (514), a force-reflective tactile manipulation sensor (516), and a tactile sensor (518).

[0181] According to another aspect of the invention, the following is provided:

[0182] D1. A remote control station (250) for controlling a robot platform (230) using multimodal sensory data (272), the remote control station (250) comprising:

[0183] A communication module (540) is used for communicatively coupling to the robot platform (230) and for receiving multimodal sensory data (272) from the robot platform (230).

[0184] Multimodal sensory data (272) includes at least two types of sensory responses; and

[0185] The user interface (610) includes an output device (612) for generating a representation of the multimodal sensory data (272) received from the robot platform (230).

[0186] D2. According to the remote control station (250) in paragraph D1, at least two sensory response types are selected from the group comprising binocular stereoscopic vision type (273a), binocular stereo audio type (273b), force-reflex tactile manipulation type (273c) and tactile type (273d).

[0187] According to another aspect of the invention, the following is provided:

[0188] E1. A method (300) for remotely controlling a robotic platform (230) in a confined space (210) of a structure (212) based on multimodal sensory data (272), the method (300) comprising:

[0189] Multimodal sensory data (272) is obtained (314) using two or more sensors (510) of the robotic platform (230).

[0190] Multimodal sensory data (272) includes at least two types of sensory responses;

[0191] Transmit (320) at least a portion of the multimodal sensory data (272) to a remote control station (250); and

[0192] Representation of multimodal sensory data (272) generated by remote control station (250).

[0193] E2. The method (300) according to paragraph E1 also provides that the robot platform (230) is positioned in the confined space (210) of the structure (212).

[0194] E3. The method (300) according to paragraph E2 also provides that, when the robot platform (230) is positioned in a confined space (210), at least a portion of the multimodal sensory data (272) is transmitted.

[0195] E4. The method (300) according to paragraph E1 also provides that at least two sensory response types are selected from the group comprising binocular stereoscopic type (273a), binocular stereo audio type (273b), force-reflex tactile manipulation type (273c) and tactile type (273d).

[0196] E5. The method (300) according to paragraph E1 also provides that it further includes adding the multimodal sensory data (272) before transmitting at least a portion of the multimodal sensory data (272).

[0197] E6. The method (300) according to paragraph E1 also provides that it further includes selecting at least a portion of the multimodal sensory data (272) for transmission.

[0198] E7. The method (300) according to paragraph E1 also provides that it further includes using the robot platform (230) to perform (350) one or more operations within a confined space (210) based on remote control instructions received at the robot platform (230) from the remote control station (250).

[0199] E8. Paragraph E7's method (300) also provides that one or more operations are selected from the following groups:

[0200] Change the position (350a) of the robot platform (230) within the confined space (210).

[0201] Drill holes (350b) in component (214) of structure (212).

[0202] Install the fastener (350c) onto the structure (212).

[0203] The structure (212) is sealed (350d).

[0204] Spray (350e) onto structure (212),

[0205] Remove the object (350f) from the confined space (210), and

[0206] Examine the structure (212) of (350g).

[0207] E9. The method (300) according to paragraph E7 also provides that the fidelity level of the multimodal sensory data (272) corresponds to one or more operations.

[0208] E10. The method (300) according to paragraph E7 also provides that the fidelity level of the multimodal sensory data (272) changes over time.

[0209] E11. The method (300) according to paragraph E7 also provides that one or more operations are performed based on local control instructions generated at the robot platform (230), such that the local control instructions are combined with remote control instructions to perform one or more operations.

[0210] E12. The method (300) provided according to paragraph E1 further includes:

[0211] Capture (440) user input at the remote control station (250); and

[0212] Remote control commands are transmitted (460) to the robot platform (230) located in a confined space (210).

[0213] E13. The method (300) according to paragraph E1 also provides that the representation of generating (430) multimodal sensory data (272) includes augmented multimodal sensory data (272) based on at least one of video spectrum, audio spectrum, spatial orientation and proprioception.

[0214] E14. The method (300) according to paragraph E1 is also provided, wherein it represents multi-sensory high-fidelity telepresence.

[0215] E15. The method (300) according to paragraph E1 also provides that the user interface (610) of the remote control station (250) includes a 3D display (613a) for presenting the binocular stereoscopic vision type (273a) in the multimodal sensory data (272).

[0216] E16. The method (300) according to paragraph A24 also provides that the user interface (610) of the remote control station (250) includes a stereo speaker (613b) for presenting the binaural stereo audio type (273b) in the multimodal sensory data (272).

[0217] Therefore, it should be understood that this disclosure is not limited to the specific examples shown, and modifications and other examples are intended to be included within the scope of the appended claims. Furthermore, while the foregoing description and associated drawings illustrate examples of this disclosure in the context of certain exemplary combinations of elements and / or functions, it should be recognized that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. Therefore, the reference numerals enclosed in parentheses in the appended claims are presented for illustrative purposes only and are not intended to limit the scope of the claimed subject matter to the specific examples provided in this disclosure.

Claims

1. A method (300) for remotely controlling a robot platform (230) based on multimodal sensory data (272), the method (300) comprising: Positioning (310) the robot platform (230), The robot platform (230) is communicatively coupled to the remote control station (250); The multimodal sensory data (272) is obtained (314) using two or more sensors (510) of the robot platform (230). The multimodal sensory data (272) includes at least two types of sensory responses; Transmit (320) at least a portion of the multimodal sensory data (272); and The robot platform (230) receives remote control commands (330) from the remote control station (250).

2. The method (300) according to claim 1, wherein the robot platform (230) is positioned in the confined space (210) of the structure (212).

3. The method (300) according to claim 2, wherein when the robot platform (230) is positioned in the confined space (210), the transmission of at least a portion of the multimodal sensory data (272) is performed.

4. The method (300) according to claim 1, wherein the at least two sensory response types are selected from the group consisting of binocular stereo vision type (273a), binocular stereo audio type (273b), force-reflex tactile manipulation type (273c) and tactile type (273d).

5. The method (300) according to claim 1, further comprising generating (334) local control commands at the robot platform (230) based on the multimodal sensory data (272).

6. The method (300) according to claim 5, further comprising using the robot platform (230) to perform (350) one or more operations within the confined space (210) based on the local control instructions.

7. The method (300) according to claim 6, wherein the multimodal sensory data (272) includes at least the binocular stereo vision type (273a), the binocular stereo audio type (273b), and the force-reflex tactile manipulation type (273c).

8. The method (300) according to claim 6, wherein the multimodal sensory data (272) includes at least the binocular stereo vision type (273a), the binocular stereo audio type (273b), the force-reflex tactile manipulation type (273c), and the tactile type (273d).

9. The method (300) according to claim 1, further comprising adding (316) the multimodal sensory data (272) before transmitting (320) the at least portion of the multimodal sensory data (272).

10. The method (300) according to claim 1, further comprising performing (350) one or more operations within the confined space (210) using the robot platform (230) based on the remote control instructions received from the remote control station (250) at the robot platform (230).

Images