Method and apparatus for mechanical arm navigation

Abstract

The invention relates to a method and apparatus for robotic arm navigation. A robotic arm is inserted into a passageway of a part to be inspected. Operator instructions, sensor readings, and an environment map defining tip motion of a tip of the robotic arm are received. The operator instructions, the environment map, and the sensor readings are applied to a previously trained machine learning model to produce a control signal. The control signal is applied to actuators on the arm to control movement of the robotic arm, allowing the robotic arm to automatically acquire traction in the passageway and move automatically according to the movement.

Description

Technical Field

[0001] These instructions generally concern the navigation of robotic arms within parts that are to be inspected, examined, processed, or maintained. Background Technology

[0002] A serpentine robotic arm is a longitudinally extending mechanical device with many degrees of freedom that can be inserted into various environments to perform inspection, maintenance, or repair. These arms typically include a large number of controlled joints that attach to cameras or other sensors inserted into the part to be inspected.

[0003] For example, a serpentine robotic arm can be inserted into an engine to inspect its internal components. The numerous degrees of freedom of these devices allow them to access messy, restricted, and / or otherwise inaccessible engine parts. Images and other information acquired by cameras can be analyzed to indicate signs of damage, wear, or other problems, while repair and maintenance activities can extend the usable life or service life of the parts. Attached Figure Description

[0004] By providing the methods and apparatus for navigation of a robotic arm as described in the following detailed description, various needs are at least partially met, particularly when studied in conjunction with the accompanying drawings. A complete and feasible disclosure of aspects of this specification, including its best mode, is set forth in the description with reference to the accompanying drawings, for those skilled in the art, wherein:

[0005] Figure 1A Including diagrams constructed based on various embodiments of these teachings;

[0006] Figure 1B Including diagrams constructed based on various embodiments of these teachings;

[0007] Figure 2 Including diagrams constructed based on various embodiments of these teachings;

[0008] Figure 3 Includes flowcharts constructed based on various embodiments of these teachings;

[0009] Figure 4 Including diagrams constructed based on various embodiments of these teachings;

[0010] Figure 5 Including diagrams constructed based on various embodiments of these teachings;

[0011] Figure 6 Including diagrams constructed based on various embodiments of these teachings;

[0012] Figure 7 Including diagrams constructed based on various embodiments of these teachings;

[0013] Figure 8Including diagrams constructed based on various embodiments of these teachings;

[0014] Figure 9 Including diagrams constructed according to various embodiments of these teachings; and

[0015] Figure 10A , 10B 10C and 10D include diagrams constructed according to various embodiments of these teachings.

[0016] The elements in the accompanying drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the size and / or relative positioning of some elements in the drawings may be exaggerated relative to other elements to aid in understanding the various embodiments of this teaching. Furthermore, common but easily understood elements that are useful or necessary in commercially viable embodiments are generally not depicted to minimize obstruction of views of these different embodiments of this teaching. Certain actions and / or steps may be described or depicted in a specific sequence of occurrence, and those skilled in the art will understand that such specificity regarding the sequence is not actually necessary. Detailed Implementation

[0017] The methods described in this paper provide extended, under-constrained mechanical snake-like arm devices with multiple degrees of freedom for use by non-specialist operators or inspectors to navigate complex spaces, such as the interior of a gas turbine engine. Operators or inspectors do not require specialized knowledge to use these devices. Simultaneously, these methods maximize the utility provided by the additional degrees of freedom of the snake-like arm to follow complex paths and adopt complex inspection postures (e.g., the lifting or positioning of a "strike cobra").

[0018] Furthermore, the method presented in this paper maximizes the robot's navigation capability through cluttered environments via self-propulsion in various traction modes (such as inchworm mode, rattlesnake mode, and bottle opener mode at the tip of a snake-arm robot), overcoming the limitations associated with pushing an endoscope from its base into a desired path without sufficient constraints to support the robot or propelling it through areas where paths are limited by friction and / or reaction forces.

[0019] In the presence of computer-aided design (CAD) models, machine learning models, or other models of the part to be tested, the navigation of snake-arm robots is further enhanced by using a task-based cost function to control the shape of the robotic arm to maximize its reach and navigation capabilities given a known final destination and / or target for the robotic arm or its tip.

[0020] In some respects, this method provides flexible snake-like robotic arms with multiple controllable degrees of freedom, but which are too slender and flexible to be self-supporting. Compared to other devices, the flexibility and additional control provide the ability to enter more complex and cluttered spaces because they are flexible and can tolerate contact with the environment, and have more support than, for example, pipe mirrors, enabling them to traverse gaps and be self-supporting over short lengths.

[0021] In other respects, the method presented in this paper enables a flexible snake-like robotic arm to employ some or all of its control degrees of freedom to pull the device from its distal end (tip) into a cluttered environment, thus eliminating the low bending stiffness along the length of the device as a limiting factor for insertion into the cluttered environment. This method also advantageously provides a user-friendly way to control such a device to follow a path, reposition the viewpoint relative to the supporting surface, and use complex motions to move forward, without requiring operator awareness or direct control to enable it.

[0022] In many such embodiments, the system includes a robotic arm, at least one sensor, an actuator, and a controller. The robotic arm is flexible and inserts into a channel within the part. The robotic arm has multiple degrees of freedom as it moves through the channel. The actuator is coupled to the robotic arm.

[0023] At least one controller may be coupled to the robotic arm, at least one sensor, and an actuator. The controller is configured to: receive operator commands defining tip movement of the robotic arm's tip; receive sensor readings from at least one sensor; access an environmental map defining the geometry of a portion of the channel; apply the operator commands, the environmental map, and the sensor readings to a previously trained machine learning model to generate control signals; and apply the control signals to the actuators on the robotic arm to control the movement of the robotic arm, thereby allowing the robotic arm to automatically acquire traction within the channel and automatically propel or move autonomously based on movement. As used herein, "traction" refers to supporting or resisting relative motion between two objects (e.g., the robotic arm and the channel). For example, traction is provided between the surfaces of the robotic arm and the channel to propel the robotic arm forward. In another example, traction is provided between the robotic arm and the surface to support the arm against gravity and / or prevent the arm from moving due to gravity.

[0024] In some instances, an environment diagram includes a computer-aided design (CAD) model. In other examples, an environment diagram includes a dynamically changing model. In still other examples, an environment diagram includes a hybrid of a CAD model and a dynamically changing model.

[0025] In this example, at least one sensor includes a camera. Other examples of sensors are possible, including but not limited to light detection and ranging (LiDAR), stereo cameras, ultrasonic sensors, and tactile sensors, to name just a few.

[0026] In other examples, the robotic arm inserts a channel at the insertion point of the part. At least one sensor includes a position sensor located at the insertion point, which determines the position of the robotic arm based on detecting markings on the robotic arm. In a further example, at least one sensor includes a camera sensor positioned to simultaneously observe the insertion point and the arm, which monitors the movement of the robotic arm based on detecting markings or specialized markings (such as QR codes or April Tags) on the robotic arm relative to a reference including the insertion point.

[0027] Various movements of the robotic arm (or parts of the robotic arm, such as its tip) are possible. For example, the movement could be that of a rattlesnake, a bottle opener, or an inchworm. Other examples are possible.

[0028] In other respects, the actuator includes or is coupled to a user interface configured to receive operator instructions from the operator.

[0029] In other embodiments, the robotic arm is inserted into a channel within the part. The robotic arm is flexible and has multiple degrees of freedom as it moves through the channel. At the controller: an operator command defining the tip movement of the robotic arm's tip is received; sensor readings from at least one sensor are received; an environment map defining the geometry of a portion of the channel is accessed; the operator command, environment map, and sensor readings are applied to a previously trained machine learning model to generate control signals; and the control signals are applied to actuators on the robotic arm to control its movement, thereby allowing the robotic arm to automatically acquire traction in the channel and automatically propel or move itself according to the movement.

[0030] The terms and expressions used herein have the same general technical meaning as those attributed to them by one of ordinary skill in the art, unless otherwise specified herein. Unless otherwise expressly stated, the word "or" as used herein should be interpreted as having a separate structure rather than a connecting structure. Unless otherwise specified herein, the terms "connection," "fixed," "attached," etc., refer to direct connection, fixation, or attachment, as well as indirect connection, fixation, or attachment via one or more intermediate components or features.

[0031] The singular forms “a,” “a,” and “the” include plural references unless the context clearly indicates otherwise.

[0032] As used herein throughout the specification and claims, approximate language is applied to modify any quantitative representation that may allow for variation without altering its associated essential function. Therefore, values ​​modified by one or more terms such as “approximately,” “about,” and “substantially” are not limited to specified exact values. In at least some cases, approximate language may correspond to the precision of the instrument used to measure the value, or the precision of the method or machine used to construct or manufacture the part and / or system. For example, approximate language may refer to a margin of 10%.

[0033] The foregoing and other benefits become clearer after a thorough review and study of the following detailed description.

[0034] Now for reference Figure 1A and Figure 1B An example of a system 100 for controlling the movement of a robotic arm 102 in a part to be measured is described. System 100 includes a robotic arm 102, an actuator 104, and a controller 106. It should be understood that the methods used can be used to insert the robotic arm 102 into a channel 103. However, these methods can also be used to control the robotic arm 102 to move out of the channel 103 in a direction substantially opposite to the insertion direction (once it has been inserted). This is a significant advantage compared to previous methods that relied on a manual extraction (e.g., pulling) device from the channel.

[0035] The robotic arm 102 is flexible and is inserted into a channel 103 within a part 108 (to be inspected, examined, processed, maintained, or machined) at an insertion point (or opening) 105. The robotic arm 102 has multiple degrees of freedom as it moves through the channel 103.

[0036] The robotic arm 102 may include a tip portion 130 (or working head) adapted to carry a sensor 132 (or tool or other inspection element) for working, inspecting or examining the channel 103, and an extended support arm 102 or structure supporting the tip portion 130. The support arm 102 includes at least one segment having a plurality of links (each link is articulated relative to its adjacent link), and means or control elements configured to control the position and / or orientation of the segment relative to a reference, thereby enabling the arm to follow and adapt to a predetermined path or shape.

[0037] This type of robotic arm is often referred to as a "serpentine arm" because it is able to move longitudinally along its own length, allowing it to follow a serpentine path. The advantage of this is that this type of arm can be used in situations where access to the work site is severely restricted. However, serpentine arm robots can also move in other modes besides moving longitudinally along a path, for example, by changing the pose and position of arm 102 without moving the base of the drive arm to make arm 102 move along the path.

[0038] In some aspects, the control of the robotic arm 102 is achieved using multiple tendons (such as ropes, or more specifically, wire ropes or cables), each rope having one end connected to a point within a plurality of links in the arm and to an actuator 104 configured to apply force and displacement at the other end of the rope. The actuator 104 is coupled to these structures and controls their movement, thereby controlling the movement and shape of the robotic arm 102. In some aspects, overlapping Bowden-cable actuation is provided to achieve wave-like motion. Wave-like motion can also be achieved without the Bowden-cable structure.

[0039] A position sensor 107 is positioned at insertion point 105 and coupled to controller 106. Position sensor 107 is configured to determine the position of robotic arm 102. For example, detecting a specific mark on robotic arm 102 as it passes position sensor 107 allows controller 106 to know how much of robotic arm 102 is within part 108. Movement of the mark on arm 102 can also allow determination of the torsional rotation of arm 102 about its axis. If the amount of robotic arm 102 within part 108 is known, the position of robotic arm 102 (and its tip) is known (or at least approximately known). In this example, position sensor 107 may be a camera that acquires a visual image processed by controller 106. Alternatively, position sensor 107 may be a sensor positioned away from robotic arm 102 and channel 103, and positioned to observe the portion of robotic arm 102 outside and simultaneously at the entrance of channel 103, such that the length of robotic arm 102 inserted into channel 103 can be determined visually. In some respects, feedback from position sensor 107 is used to obtain corkscrew motion.

[0040] The actuator 104 may be a motor or other device that actuates the wire rope or cable of the robotic arm 102 and also actuates the robotic arm 102. In some cases, the robotic arm 102 is pushed by a person (e.g., the robotic arm 102 is initially inserted into part 108).

[0041] In some other examples, actuator 104 receives instructions to define a target (e.g., a destination within a channel) for the tip portion 130 of robotic arm 102. In these respects, actuator 104 may include user interface 121 to allow an operator to input commands specifying a destination. For example, user interface 121 may include a joystick. In another example, the final destination is unknown at the start of operation, and the operator can use user interface 121 to interactively manipulate robotic arm 102 in exploratory mode. Other examples of operator input are possible.

[0042] An environment diagram 111 (e.g., a computer-aided design (CAD) model or a dynamically generated drawing) is stored in memory 110. Memory 110 can be any type of electronic storage device. Environment diagram 111 includes information relating to channel 103 and part 108. For example, environment diagram 111 specifies the shape, size, relative positioning, and / or relative position of channel 103, obstacles in channel 103, and the shape, size, relative positioning, and / or relative position of internal components of part 108. For example, if part 108 is an aircraft engine, the position, shape, size, relative positioning, and / or relative position of the engine's channels, blades, impellers, and rotor can be included in environment diagram 111.

[0043] Controller 106 is coupled to robotic arm 102 (and sensor 132), position sensor 107, and actuator 104. In some aspects, controller 106 is located at or within actuator 104. It should be understood that, as used herein, the term “controller” refers broadly to any microcontroller, computer, or processor-based device having a processor, memory, and programmable input / output peripherals, which is generally designed to manage the operation of other components and devices. It should also be understood to include common accessory devices, including memory, transceivers for communicating with other components and devices, etc. These architectural options are known and understood in the art and need not be described further herein. Controller 106 may be configured (e.g., by using a corresponding program stored in memory, as understood by those skilled in the art) to perform one or more of the steps, actions, and / or functions described herein.

[0044] Memory 110 also stores a machine learning model 115. Examples of machine learning models include neural networks and other such structures. If the machine learning model 115 is a neural network, the neural network can be trained. In examples, training may include broadcasting or incorporating movements into the neural network. For example, sidewinder movement, corkscrew movement, and inchworm movements may be incorporated into the neural network. In another example, the machine learning model 115 is trained by having a robotic arm pass through channels in a part (e.g., a part used for training or testing), causing the machine learning model to learn certain behaviors (e.g., performing a specific action when encountering an obstacle).

[0045] The machine learning model 115 can be trained in a supervised or unsupervised manner. Supervised algorithms select targets or desired outcomes predicted from a given set of predictors (independent variables). Using this set of variables, a function or structure is generated that maps the inputs to the desired output. The training process continues until the machine learning model 115 achieves the desired level of accuracy on the training data. Examples of supervised learning algorithms include regression, decision trees, random forests, k-nearest neighbors (KNN), and logistic regression. In some aspects, supervised learning can use labeled data. In one example, the desired output includes desired control signals to achieve the desired outcome.

[0046] In unsupervised learning, no objective is used. Instead, these methods group the population into different clusters based on patterns. Examples of unsupervised learning methods include the Apriori algorithm and the K-means method.

[0047] exist Figure 1A In one example of the system's operation, controller 106 is configured to receive instructions (e.g., operator input 126) and access environment graph 111 from memory 110. Based on the instructions and environment graph 111, controller 106 applies these to machine learning model 115 to obtain one or more control signals 120. The one or more control signals 120 are applied to actuator 104 to change the configuration state of robotic arm 102 using multiple degrees of freedom, thereby allowing the robotic arm to obtain traction in channel 103 and propel itself toward its destination based on movement not involving actuator 104. As described above, these state changes can be implemented by actuator 104.

[0048] The shape-changing properties of the robotic arm 102 allow it to acquire traction within the channel 103 and propel itself through it. For example, an inchworm-like movement made by the robotic arm 102 causes a portion of the robotic arm 102 to grip and push against the bottom of the channel 103, thereby propelling all portions of the robotic arm 102 forward. Obstacles in the channel 103 can be used as levers for the robotic arm 102 to encourage and facilitate its self-propelled movement. In doing so, it can be seen that once the operator inserts the robotic arm 102 into the part 108, no operator intervention is required. It should also be understood that while the methods described herein allow the robotic arm 102 to propel itself through at least some portions of the channel 103, the actuator 104 can provide additional force or power to move the robotic arm 102 through the channel 103. It should also be understood that the resulting movement is not limited to the specific movements mentioned herein (e.g., rattlesnake, bottle opener, or inchworm), but can be combinations or variations of these movements, among other entirely different movements.

[0049] In some cases, actuator 104 does not push robotic arm 102. Instead, a person pushes (and may twist) robotic arm 102 and position sensor 107 measures the displacement of robotic arm 102 and treats the sensed displacement as a control input.

[0050] In some respects, the robotic arm includes a distal end (including the tip portion 130) and a proximal end (opposite to the distal end). The tip portion 130 (including the tip) of the robotic arm 102 is controlled so that it does not necessarily come into contact with support from the environment (channel 103). Using multiple degrees of freedom of the robotic arm 102, most of the distal environmental support for the distal end of the robotic arm 102 (including the tip portion 130) can be provided by the channel 103 toward the proximal end of the robotic arm 102. For example, support for the robotic arm 102 can be located at a second or third segment of the robotic arm 102 behind the tip portion 130. In operation, the operator will not be aware that the machine learning model 115 is controlling more portions of the robotic arm 102 so that the tip portion 130 (including the tip) can be raised further away from any support surface in the channel 103. In one example, the tip portion 130 includes the first two segments of the robotic arm 100 located at the distal end.

[0051] Now for reference Figure 1B The operation of system 100 is further described below. Dynamic drawing generation method 122 or CAD method 124 can be used to create environment drawing 111. Alternatively, a combination of these two methods can be used.

[0052] CAD method 124 may include a CAD file that describes part 108 (e.g., channel 103) using an environment diagram 111. The CAD file may be in any suitable format.

[0053] The dynamic map generation method 122 can operate according to a Simultaneous Localization and Mapping (SLAM) method. The SLAM method constructs an environmental map of part 108 (e.g., an aircraft engine) by using sensor inputs (e.g., sensor 132) as the robotic arm 102 moves past part 108 (e.g., through channel 103). The environmental map 111 created by the SLAM method (or other dynamic method) changes dynamically over time. For example, the environmental map 111 may initially include very little information, then be filled with more detailed information over time. Using such a dynamic method effectively adapts the environmental map 111 to part 108 (i.e., a specific part, not just a part of a given type). For example, if part 108 is an aircraft engine, the blades in aircraft engine A may be unbent from their nominal shape, but the blades in aircraft engine B may be bent from their nominal shape. Additionally, the blades of engine A may be unbent at a first time but bent at a subsequent time. In these examples, separate environmental maps are constructed for engine A and engine B respectively, and a separate environmental map is constructed for engine A at the first and second times.

[0054] Environmental diagram 111, operator input 126 (e.g., joystick position), and sensor input 128 (e.g., readings from position sensor 107) are applied to machine learning model 115 to generate one or more control signals 120. The one or more control signals 120 are applied to actuator 104. Actuator 104 uses these to control robotic arm 102. In doing so, the movement of robotic arm 102 is controlled, allowing robotic arm 102 to automatically acquire traction in channel 103 and automatically propel itself according to the movement.

[0055] As an example, when robotic arm 102 moves through channel 103, it may encounter an obstacle within channel 103. An environmental diagram 111, operator input 126 (e.g., joystick position), and sensor input 128 (e.g., readings from position sensor 107) are applied to machine learning model 115. Machine learning model 115 determines that inchworm movement will allow robotic arm 102 to overcome the obstacle. A control signal 120 inducing this movement is sent to actuator 104, which causes robotic arm 102 (or a portion of robotic arm) to perform the specified movement. The operator is only aware that robotic arm 102 is moving forward and does not need to steer to handle the obstacle. In other words, overcoming the obstacle is handled automatically by system 100.

[0056] Now for reference Figure 2 This describes an example of a system 200 for navigating a robotic arm 202 with a tip 201 in a part 204 to be inspected. The tip 201 may include a first sensor (e.g., a camera, not shown, see [link]). Figure 1A Sensor 132 in the middle.

[0057] The robotic arm 202 includes a first segment 206 and a second segment 208, the first segment 206 being a passive arm segment for manual manipulation, and the second segment 208 including a plurality of actuation segments 210. In the example, a person can push the robotic arm 202. The actuator assembly 212 includes motors, gears, and other mechanisms to push and control the movement of the robotic arm 202. The robotic arm 202 may include ropes, cords, cables, gears, pulleys, or mechanisms that, when adjusted in a particular manner by the actuator assembly 212, cause the robotic arm 202 to propel itself and cause that movement to take on various shapes (such as rattlesnake motion, bottle opener motion, or inchworm motion, to name a few). The actuator assembly 212 may also include a controller (e.g., controller 106) to perform the various operations described herein. As described elsewhere herein, deep reinforcement learning can be used to obtain a machine learning model 115. The machine learning model 115 is seeded or trained with these behaviors as a starting point.

[0058] The part to be inspected 204 includes a pipe mirror inspection (BSI) port 205 that allows access to a passage 207 within the part to be inspected 204. For clarity, the upper portion of the part to be inspected 204 has been removed to allow view of a portion of the robotic arm 202 within the part to be inspected 204. The part to be inspected 204 may be an aircraft engine, and the passage 207 may be located inside the engine.

[0059] In some respects, the second (or position) sensor 214 is mounted on or near the BSI port 205. The robotic arm 202 is initially passive until its tip 201 is sensed by the position sensor 214 when inserted by the operator. The second sensor 214 uses markings on the exterior of the robotic arm 202 to sense the position (e.g., depth of insertion channel 207) and rotation of the robotic arm 202 when it is manually inserted. Snake-like motion is subordinate to insertion sensing, and actuator motion is similarly subordinate to insertion sensing, to maintain the passive segment 206 of the robotic arm 202 in a maneuverable condition rather than being pushed or pulled. Other types of sensors may also be used. For example, magnetic properties may be encoded onto the skin of the robotic arm 202, and the second sensor 214 may be a Hall effect sensor that senses these properties.

[0060] In other respects, the second sensor 214 houses a stabilizing gripping element. This stabilizing gripping element (e.g., a clamp or gripper) allows the robotic arm 202 to remain stationary for single-handed operation when the operator needs to keep the arm still. Intention is sensed via arm skin, visual gesture sensing, or when explicitly indicated by a switch (e.g., a foot pedal switch). In other embodiments, the stabilizing gripping element may be separate from or adjacent to the second sensor 214.

[0061] Control of the robotic arm 202 can be accomplished by an operator through manual insertion. In this case, the operator inserts the robotic arm 202, much like inserting a pipe mirror, and controls the direction of movement or tip joint connections (e.g., up / down / left / right) using a joystick (e.g., user interface 121). The operator can manipulate the tip 201 using images obtained from a camera (e.g., see sensor 132) mounted at the tip. Although the camera image can be software-rotated for display, the operator is generally familiar with twisting the robotic arm 202, so the arm twist of the robotic arm 202 is sensed by the second sensor 214 and enabled as a usage mode. In some embodiments, the camera image may be software-rotated such that the vertical axis of the image is aligned with the gravity vector, while in other embodiments, the camera image is not rotated, and the arm twist of the robotic arm 202 causes the image to rotate.

[0062] In other respects, a first sensor (e.g., sensor 132) at the tip 201 is used to identify features in the insertion environment in real time to generate an environmental model for assessment of support, stabilization, gap bridging, arm constraints, and limitations. The operator focuses only on tip movement based on vision-based (e.g., camera) feedback, while the control software uses multiple degrees of freedom in the arm to achieve stable forward movement and desired tip position, and traction movement where possible.

[0063] In some cases, CAD models can be used for the environment (e.g., channel 207) and can constrain task-based cost functions to achieve navigation assistance. Given a final goal, these task-based cost functions help to optimally bias the arm shape of the robotic arm 202 without limiting operator interaction. Typical cost functions may include the sum of the magnitudes of the bending angles between the degrees of freedom of the arm, as a global measure of the joint connections of the robotic arm 202, or the sum of the squares of the bending angles, in order to minimize the extreme values ​​of the bending angles between the degrees of freedom of the robotic arm 202.

[0064] Now for reference Figure 3 This describes an example of a method for controlling a robotic arm.

[0065] In step 302, a robotic arm is inserted into the channel of the part to be inspected. The robotic arm is flexible and includes a tip with a sensor at the tip. The robotic arm has multiple degrees of freedom as it moves through the channel.

[0066] In step 304, the environment map is accessed. The environment map may be stored in electronic memory. The environment map may be static (e.g., a CAD model) or dynamically changing (e.g., generated and updated using SLAM methods).

[0067] In step 306, sensor readings are received. These can come from the tip of the robotic arm. Alternatively, these readings can come from sensors at the input port.

[0068] In step 308, user commands are received. These user commands can come from a joystick. Other examples are also possible.

[0069] In step 310, the environmental map, sensor readings, and user instructions are applied to the machine learning model to generate control signals. As discussed elsewhere in this document, the machine learning model can be any type of machine learning model, such as a neural network.

[0070] In step 312, a control signal is applied to the actuator of the arm or robotic arm to move the robotic arm.

[0071] Now for reference Figure 4 , Figure 5 and Figure 6 Examples of complex movements provided by this method are described. It should be understood that these movements can cause movement of all or part of the arm (e.g., the portion of the arm around or behind the tip). Furthermore, combinations of these movements are possible. Even further, other complex movements are possible.

[0072] Now for reference Figure 4 This describes an example of a robotic arm 402 (or a portion thereof) that moves in a rattlesnake-like motion. Figure 4 The movement of robotic arm 402 is shown, generally in the direction indicated by the arrow marked 420. As shown, sections of robotic arm 402 are actuated to push the robotic arm 402. These sections push objects (e.g., the sides of a passage, obstacles, internal components of the part being inspected). The pushing action of certain sections of robotic arm 402 causes the entire robotic arm 402 to move in the direction indicated by the arrow marked 420, and causes robotic arm 402 to adopt a rattlesnake shape.

[0073] More specifically, robotic arm 402 first takes shape 430. Segments 408 and 410 are actuated to push robotic arm 402 in the direction indicated by the arrow marked 406, which moves robotic arm 402 and causes robotic arm 402 to take shape 432. Then segments 412 and 414 are actuated to push robotic arm 402 in the direction indicated by the arrow marked 406, which moves robotic arm 402 and causes robotic arm 402 to take shape 434. Once robotic arm 402 is in shape 434, segments 416 and 418 can be pushed in the direction indicated by the arrow marked 406 to move robotic arm 402 in direction 420 and also take on a different shape.

[0074] It should be understood that this is one example of rattlesnake movement and other examples are possible. It should also be understood that the movement of the arm in direction 420 can be relatively large or small. Furthermore, in other respects, there is no movement in direction 420, and the rattlesnake motion merely changes the shape of the robotic arm 402.

[0075] Now for reference Figure 5 This describes an example of the bottle opener motion of a robotic arm 500. Sections of the robotic arm 500 are actuated to move the robotic arm 500 forward in the direction indicated by the arrow marked 504. Actuation of these sections can also occur in the direction indicated by the arrow marked 502. Similarly, by actuating selected sections of the robotic arm 500, the shape of the robotic arm 500 is made to resemble a bottle opener.

[0076] Now for reference Figure 6 This describes an example of the inchworm movement of the robotic arm 602. In state 630, the robotic arm 602 is straight, for example, lying upright in the channel. Next, in state 632, a portion of the robotic arm 602 is actuated downwards (e.g., against the bottom of the channel), pushed in the direction indicated by the arrow marked 604, causing the robotic arm 602 to bend upwards, and also moving the robotic arm 602 forward in the direction indicated by the arrow marked 606.

[0077] Then, in state 634, the other parts of robotic arm 602 are actuated downwards by a push in the direction indicated by the arrow marked 604 (e.g., against the bottom of the channel), causing robotic arm 602 to bend further upwards and also move further forward in the direction indicated by the arrow marked 606. Next, in state 636, the pressure from the parts of robotic arm 602 is released, causing robotic arm 602 to bend downwards and also move further forward in the direction indicated by the arrow marked 606. Finally, in state 638, robotic arm 602 is straight and has moved forward relative to state 630.

[0078] Now for reference Figure 7 This describes an example of environment diagram 700 (e.g., environment diagram 111). Figure 7This is a visualization of Environment Diagram 700, and it should be understood that this can be represented electronically in any type of format. For example, Environment Diagram 700 can be a computer-aided design (CAD) file in any suitable format that describes the environment in which the robotic arm will operate. In another example, Environment Diagram 700 is a file created according to the SLAM method. If it is a CAD file, Environment Diagram 700 can be static. However, if it is created according to the SLAM (or other similar) method, Environment Diagram 700 will change dynamically over time, for example, from containing less information about the environment to containing more information about the environment in which the robotic arm will operate.

[0079] In either case, and in this example, the environment diagram 700 includes information about a channel 702 having an obstacle 706 leading to a blade 704. Dimension D1 is the distance from the opening to the obstacle 706. Distance D2 is the length of the blade 704. Theta is the angle of the obstacle 706 relative to the blade 704. D3 is the distance from the opening of the channel to the tip of the blade 704. D4 is the height of the obstacle 706.

[0080] All this information is included in the appropriate format to represent the environment. As mentioned, the environment diagram 700 can physically be a file of any format, which contains a representation of the environment (in this case, Figure 7 Any appropriate data structure for the environment shown in the diagram.

[0081] Now for reference Figure 8 An example of a machine learning model 800 is described. The machine learning model 800 includes various interconnected nodes 808, 810, 812, 814, 816, 818, 820, 822, 824, 826, 828, 830, 832, and 834 as shown in the figure. An environment map 802 is applied to node 808. Sensor readings 804, for example, from a position sensor (e.g., position sensor 214), are applied to node 810. User input (e.g., joystick movement) is applied to node 812. As a result of receiving a specific environment map 802, a specific sensor reading 804, and a specific user input 806, the machine learning model 800 has been trained to generate specific and distinct control signals 836. In this example, the machine learning model 800 is a convolutional neural network. In other examples, the machine learning model 800 uses deep reinforcement learning. Different training sets simulating user input and sensor readings can be applied.

[0082] In one example, machine learning model 800 is a neural network, and machine learning model 800 is trained by applying training data to the neural network. In a specific example, machine learning model 800 is as follows: Figure 8The neural network shown has layers 850, 852, 854, and 856 (each containing one or more nodes), and each layer performs one or more specific functions. In some respects, these layers form a graphical structure of vectors or matrices with weights having specific values. For example, layer 850 may be an input layer that receives input signals or data and passes that information to the next layer 852. One or more other layers 852, 854 perform calculations or make determinations on the data or related data. Layer 856 may be an output layer that generates a control signal 836 that causes arm movement.

[0083] Now for reference Figure 9 This describes an example of the functional operation of a machine learning model (e.g., machine learning model 115). In step 902, various inputs are received. In this case, the environment map input indicates obstacles in the path of the robotic arm and the size (X) of the obstacles. Sensor readings indicate the position (Y) of the robotic arm in the channel. User input indicates that the user wishes to move the robotic arm forward (e.g., further into the channel from position Y). In step 904, these inputs are applied to the machine learning model (e.g., a trained neural network) and, through various calculations and analyses, it is determined that the action to be performed is to move a portion of the arm over the obstacle using inchworm-like movement. Therefore, a control signal 906 is formed to be applied to the robotic arm, thereby forcing a portion of the arm into an inchworm-like movement that will overcome the obstacle. It should be understood that this action is determined and performed without the operator's knowledge or intervention and can be transparent to the operator.

[0084] Now for reference Figure 10A , Figure 10B , Figure 10C and Figure 10D This describes an example of controlling a robotic arm 1002. As previously stated, "traction" as used herein refers to supporting or resisting relative motion between two objects (e.g., the robotic arm and a channel). For example, traction is provided between the surfaces of the robotic arm and the channel to propel the robotic arm forward. In another example, traction is provided between the robotic arm and the surface to support the arm against gravity and / or prevent the arm from moving due to gravity.

[0085] It should be understood that the methods used can be used to insert the robotic arm 1002 into the channel. However, these methods can also be used to control the robotic arm 1002 to move out of the channel in a direction substantially opposite to the insertion direction (once it has been inserted). As previously stated, this is a significant advantage over previous methods, which relied on a manual extraction (e.g., pulling) device from the channel.

[0086] The robotic arm 1002 moves along the channel surface 1004. The robotic arm includes a first portion 1006, a second portion 1008, a third portion 1010, and a fourth portion 1012. Each of the first portion 1006, the second portion 1008, the third portion 1010, and the fourth portion 1012 includes one or more segments that are actuated as described elsewhere herein. Segment 1012 has a tip 1014, which is the distal end of the robotic arm 1002. Segment 1006 is located at the proximal end of the robotic arm 1002.

[0087] exist Figure 10A In the middle, the operator points the tip 1014 upwards to view targets or features (e.g., inside the aircraft engine). Figure 10B In the middle, the operator has already moved the robotic arm 1002 forward in the approximate direction indicated by the arrow marked 1016 (with...). Figure 10A (In comparison) and jointed, thus allowing the target or feature to be viewed at a smaller angle relative to the horizontal plane. So far, the movement of all segments has been "path following" (tip following), as all segments move in the same way as tip 1014.

[0088] However, in Figure 10C In the middle, the operator has moved further forward and the joint connection downwards a little more. However, the second segment 1008 of the robotic arm 1002 does not bend to follow the direction of the connection. Figure 10B The path described by the robotic arm 1002. The traction provided by the second segment 1008 allows the third segment 1010 and the fourth segment 1012 (together with the tip 1014) to be raised and angled without the third segment 1010 or the fourth segment 1012 needing to obtain traction.

[0089] Then, in Figure 10D In this process, the operator again moves the robotic arm 1002 forward to a low level (to the horizontal plane). Throughout this movement, the operator is unaware of the buckling of the robotic arm 1002 behind the fourth segment 1012, which is necessary to achieve the angle of the tip 1014 and the elevation of the fourth segment 1012. The operator is also unaware of the positioning and orientation of the fourth segment 1012 near its proximal end. Positioning occurs automatically using a machine learning model, without operator intervention or control, as described elsewhere in this document.

[0090] Further aspects of this disclosure are provided by the subject matter of the following clauses:

[0091] A system comprising: a robotic arm, the robotic arm being flexible, inserted into a channel within a part, the robotic arm having multiple degrees of freedom when moving through the channel; at least one sensor; an actuator coupled to the robotic arm; and a controller coupled to the robotic arm, the at least one sensor, and the actuator, the controller being configured to: receive an operator command defining a tip movement of a tip of the robotic arm; receive sensor readings from the at least one sensor; access an environment map defining the geometry of a portion of the channel; apply the operator command, the environment map, and the sensor readings to a previously trained machine learning model to generate a control signal; and apply the control signal to the actuator on the robotic arm to control movement of the robotic arm, thereby allowing the robotic arm to automatically acquire traction in the channel and move automatically within the channel according to the movement.

[0092] The system according to any of the foregoing clauses, wherein the environment diagram includes a computer-aided design (CAD) model.

[0093] The system according to any of the foregoing clauses, wherein the environment map includes a dynamically changing model.

[0094] According to any of the foregoing clauses, the environment diagram comprises a hybrid of computer-aided design (CAD) models and dynamically changing models.

[0095] The system according to any of the foregoing clauses, wherein the at least one sensor includes a camera.

[0096] According to any of the preceding clauses, the robotic arm inserts into the channel at the insertion point of the part, and wherein the at least one sensor includes a position sensor located at the insertion point, the position sensor determining the position of the robotic arm based on detecting a mark on the robotic arm.

[0097] According to any of the foregoing clauses, the movement includes rattlesnake movement.

[0098] According to any of the foregoing clauses, the movement includes bottle opener movement.

[0099] According to any of the foregoing clauses, the movement includes inchworm movement.

[0100] The system according to any of the foregoing clauses, wherein the actuator includes a user interface configured to receive operator instructions from the operator.

[0101] According to any of the foregoing clauses, the robotic arm includes a tip portion having at least one segment and a proximal portion having multiple segments, wherein selected segments of the multiple segments of the proximal portion provide traction to lift and / or change the orientation of the tip portion, while the tip portion does not receive traction from the channel and there is no operator involvement or control.

[0102] A method comprising: inserting a robotic arm into a channel within a part, the robotic arm being flexible and having multiple degrees of freedom when moving through the channel; at a controller: receiving an operator instruction defining tip movement of a tip of the robotic arm; receiving sensor readings from at least one sensor; accessing an environment map defining the geometry of a portion of the channel; applying the operator instruction, the environment map, and the sensor readings to a previously trained machine learning model to generate a control signal; and applying the control signal to actuators on the robotic arm to control movement of the robotic arm, thereby allowing the robotic arm to automatically acquire traction in the channel and move automatically within the channel according to the movement.

[0103] The method according to any of the foregoing clauses, wherein the environment diagram includes a computer-aided design (CAD) model.

[0104] The method according to any of the foregoing clauses, wherein the environment map includes a dynamically changing model.

[0105] According to any of the foregoing clauses, the environment diagram comprises a hybrid of a computer-aided design (CAD) model and a dynamically changing model.

[0106] The method according to any of the foregoing clauses, wherein the at least one sensor includes a camera.

[0107] According to any of the preceding clauses of the method, the robotic arm is inserted into the channel at the insertion point of the part, and wherein the at least one sensor includes a position sensor located at the insertion point, the position sensor determining the position of the robotic arm based on detecting a mark on the robotic arm.

[0108] The method according to any of the foregoing clauses, wherein the movement includes rattlesnake movement.

[0109] The method according to any of the foregoing clauses, wherein the movement includes movement of the bottle opener.

[0110] The method according to any of the foregoing clauses, wherein the movement includes inchworm movement.

[0111] The method according to any of the foregoing clauses, wherein the actuator includes a user interface configured to receive operator instructions from the operator.

[0112] According to any of the foregoing clauses, the robotic arm includes a tip portion having at least one segment and a proximal portion having multiple segments, wherein selected segments of the multiple segments of the proximal portion provide traction to lift and / or change the orientation of the tip portion, while the tip portion does not receive traction from the channel and there is no operator involvement or control.

[0113] Those skilled in the art will recognize that various modifications, variations, and combinations can be made to the above embodiments without departing from the scope of this disclosure, and such modifications, variations, and combinations should be considered within the scope of the inventive concept.

Claims

1. A system for controlling the movement of a robotic arm during the inspection, maintenance, or repair of parts, characterized in that, The system includes: A robotic arm, which is flexible, is inserted into a channel within a part, and has multiple degrees of freedom when moving through the channel; At least one sensor; An actuator, which is coupled to the robotic arm; A controller, connected to the robotic arm, the at least one sensor, and the actuator, is configured to: Receive operator commands that define the tip movement of the robotic arm's tip; Receive sensor readings from the at least one sensor; Access environment map, the environment map defining the geometry of a portion of the channel; The operator instructions, the environmental map, and the sensor readings are applied to a previously trained machine learning model to generate control signals; and The control signal is applied to the actuator of the robotic arm to control the movement of the robotic arm, thereby allowing the robotic arm to automatically acquire traction in the channel and move automatically within the channel according to the movement.

2. The system according to claim 1, characterized in that, in, The robotic arm includes a tip portion having at least one segment and a proximal portion having multiple segments, wherein selected segments of the multiple segments of the proximal portion provide traction to lift and / or change the orientation of the tip portion, while the tip portion does not receive traction from the channel and is not operated or controlled by an operator.

3. The system according to claim 1, characterized in that, in, The environmental diagram includes a computer-aided design (CAD) model or a dynamically changing model.

4. The system according to claim 1, characterized in that, in, The environment diagram is a hybrid of computer-aided design (CAD) models and dynamically changing models.

5. The system according to claim 1, characterized in that, in, The at least one sensor includes a camera.

6. The system according to claim 1, characterized in that, in, The robotic arm is inserted into the channel at the insertion point of the part, and wherein the at least one sensor includes a position sensor located at the insertion point, the position sensor determining the position of the robotic arm based on detecting a mark on the robotic arm.

7. The system according to claim 1, characterized in that, in, The movement includes rattlesnake movement.

8. The system according to claim 1, characterized in that, in, The movement includes the movement of the bottle opener.

9. The system according to claim 1, characterized in that, in, The movement includes inchworm movement.

10. The system according to claim 1, characterized in that, in, The actuator includes a user interface configured to receive operator instructions from the operator.

11. A method for navigating a robotic arm, characterized in that, The method includes: A robotic arm is inserted into a channel within the part; the robotic arm is flexible and has multiple degrees of freedom when moving through the channel. At the controller: Receive operator commands that define the tip movement of the robotic arm's tip; Receive sensor readings from at least one sensor; Access environment map, the environment map defining the geometry of a portion of the channel; The operator instructions, the environment map, and the sensor readings are applied to a previously trained machine learning model to generate control signals; The control signal is applied to the actuator on the robotic arm to control the movement of the robotic arm, thereby allowing the robotic arm to automatically obtain traction in the channel and move automatically within the channel according to the movement.

12. The method according to claim 11, characterized in that, in, The robotic arm includes a tip portion having at least one segment and a proximal portion having multiple segments, wherein selected segments of the multiple segments of the proximal portion provide traction to lift and / or change the orientation of the tip portion, while the tip portion does not receive traction from the channel and is not operated or controlled by an operator.

13. The method according to claim 11, characterized in that, in, The environmental diagram includes a computer-aided design (CAD) model or a dynamically changing model.

14. The method according to claim 11, characterized in that, in, The environment diagram is a hybrid of computer-aided design (CAD) models and dynamically changing models.

15. The method according to claim 11, characterized in that, in, The at least one sensor includes a camera.

16. The method according to claim 11, characterized in that, in, The robotic arm is inserted into the channel at the insertion point of the part, and wherein the at least one sensor includes a position sensor located at the insertion point, the position sensor determining the position of the robotic arm based on detecting a mark on the robotic arm.

17. The method according to claim 11, characterized in that, in, The movement includes rattlesnake movement.

18. The method according to claim 11, characterized in that, in, The movement includes the movement of the bottle opener.

19. The method according to claim 11, characterized in that, in, The movement includes inchworm movement.

20. The method according to claim 11, characterized in that, in, The actuator includes a user interface configured to receive operator instructions from the operator.

Images