A high degree of freedom robot working system based on remote visual feedback control

Abstract

The application provides a high-degree-of-freedom robot operation system based on remote visual feedback control, which comprises a field device end and a remote control end connected with the field device end through a picture transmission communication link; the field device end comprises a field operation robot and a field all-range omnidirectional visual device; the field operation robot comprises a multi-stage mechanical arm and an operation arm installed at the end of the multi-stage mechanical arm; the application installs the all-range omnidirectional visual device on the multi-stage mechanical arm and the operation arm, generates multiple visual points, passes through the dead zones and dead corners of a single visual point, and gradually combines a complete visual channel in the obstacle space, so that the visual dead zones and dead corners of a single visual point in a complex obstacle space are avoided; the joints of the multi-stage mechanical arm are gradually moved in control, the all-range omnidirectional obstacle avoidance sensing and all-range omnidirectional obstacle avoidance movement of the field operation robot in a complex obstacle scene are ensured, and the operation task is completed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of robot operation system, and particularly relates to a high-degree-of-freedom robot operation system based on remote visual feedback control. BACKGROUND

[0002] Under the harm to the environment and the complex operation demand, remote manual control can utilize the skills of the staff itself, can not be on the scene, and can complete the complex operation which cannot be completed by the present artificial intelligence and fixed program through unmanned equipment. The remote manual control unmanned operation technology has achieved many results in the fields of industrial production, medical health, disaster relief support and the like.

[0003] The existing remote operation system works in an unobstructed space, such as air, space, water bottom, unobstructed table, factory building. In a typical mechanical arm working scene, the mechanical arm is fixedly installed on a base, the base and a workpiece are on a working surface, the mechanical arm moves in a vertical space above the working surface and between the base and the workpiece, and the moving space of the mechanical arm is unobstructed or basically unobstructed, that is, the moving space of the mechanical arm is an unobstructed space in which the mechanical arm can freely move. From the base of the mechanical arm to the workpiece, the mechanical arm generally extends in a straight line to reach the workpiece in the shortest path, or bends once or a few times to extend the arm to reach the workpiece. The joint design with multiple degrees of freedom is mainly used for the mechanical arm to move to different workpiece positions, and the camera for remote sensing is generally placed above the working surface of the mechanical arm or at the end of the mechanical arm, so as to provide a remote operator with a view of the operation.

[0004] For example, CN201910996207.1 provides an industrial robot vision system, which realizes remote visual feedback by installing a vision module at the front end of the robot and connecting a laser vision sensor at the end of the operation manipulator. For example, CN202211392795.6 realizes remote control of the mechanical arm in the model space by modeling and reproducing the state of the mechanical arm in VR through a three-dimensional information acquisition camera of the mechanical arm. For example, WO2008109993A1 and EP2134606B1 disclose a satellite robot refueling scheme, which realizes remote control of the mechanical arm to complete the refueling work through the camera of the focusing mechanical arm and the end effector.

[0005] But in a complex obstacle space, there are multiple obstacles on the movement plane from the base of the robot arm to the workpiece, which cannot be directly reached, and the robot arm needs to be bent multiple times to avoid obstacles, and only in a complex obstacle space can it be reached by multiple-plane "snake-like" twisting and crossing. However, the aforementioned CN201910996207.1 only installs security cameras at the front end of the robot movement and the end of the work robot arm, and does not consider the obstacle avoidance of the robot and the robot arm in a complex environment; CN202211392795.6 installs a three-dimensional information acquisition camera containing the entire robot arm to model the scene and the robot arm in VR, but there are problems such as insufficient modeling accuracy, insufficient completeness, and low timeliness of modeling due to the large amount of calculation required for VR scene modeling; The scene of the scheme of WO2008109993A1 and EP2134606B1 is in space, and a camera is installed on the focusing robot arm and the end effector. The space scene is spacious, and the design does not consider the obstacle avoidance of the robot arm itself.

[0006] Therefore, how to achieve full-range omnidirectional on-site sensing and obstacle avoidance and high-precision work in a complex obstacle space for a large load and a super-long high-degree of freedom, so as to replace traditional manual on-site work, is a problem to be solved. SUMMARY

[0007] The present application aims to solve the technical problems existing in the prior art, and the purpose of the present application is to provide a high-degree-of-freedom robot work system based on remote visual feedback control.

[0008] To achieve the above purpose, the present application adopts the following technical scheme: a high-degree-of-freedom robot work system based on remote visual feedback control, comprising a field device end and a remote control end connected to the field device end through image transmission communication link. Remote operators observe the work state, movement state, and obstacle state of the surrounding work space of the field device end through remote omnidirectional vision at the remote control end, and feedback control the movement and work of the field device end; the field device end includes a field work robot and a field omnidirectional vision device, the field work robot includes a base, a multi-stage robot arm installed on the base, and a work arm installed at the end of the last stage robot arm, and the field omnidirectional vision device is installed on the work arm and each stage or part of the robot arm, and the field omnidirectional vision device provides the required omnidirectional view for the work arm and the robot arm.

[0009] The technical scheme above installs an all-directional vision device on the multi-stage mechanical arm and the operation arm throughout the whole process, generates multiple visual points, and forms a complete visual channel in the obstacle space by traversing the dead zones and blind spots of the single visual point, thereby avoiding the visual dead zones and blind spots of the single visual point in the complex obstacle space; the joints of each stage of the mechanical arm are moved step by step in control, to ensure the all-directional obstacle avoidance sensing and all-directional obstacle avoidance movement of the field operation robot in the complex obstacle scene, and the operation task is completed. The remote control end of the application can capture the field of view of all movable ranges of the field device end in a control period, and through the image transmission communication link between the field device end and the remote control end, the remote operator can use the system to remotely operate accurately and safely in various complex environments.

[0010] In a preferred embodiment of the application, each stage of the mechanical arm comprises a transverse rotation joint, an extended rear arm, a longitudinal rotation joint and an extended front arm connected in sequence, and each stage of the mechanical arm can be longitudinally stretched or folded through the longitudinal rotation joint, the extended rear arm and the extended front arm, and the multi-stage mechanical arm can be twisted and "snaked" in the three-dimensional operation space through the transverse rotation joint.

[0011] The technical scheme above enables the multi-stage mechanical arm to twist and "snake" in the three-dimensional operation space, to realize multi-plane twisting and traversing in the complex obstacle space, thereby realizing obstacle avoidance in the complex three-dimensional obstacle space, and finally realizing multi-plane "snaking" type twisting and traversing to the operation position in the complex obstacle space, and then performing operation through the operation arm.

[0012] In a preferred embodiment of the application, the operation arm comprises an upper arm connected to the end of the last stage of the mechanical arm through a shoulder joint, a lower arm connected to the upper arm through an elbow joint, and a mechanical hand connected to the lower arm through a wrist joint.

[0013] The technical scheme above adopts a structure similar to the human arm, so that the human body sensing device or the human body motion visual recognition device can be used at the remote control end, and the remote operator can control the operation arm to perform field operation through the movement of the arm, which is intuitive and efficient.

[0014] In a preferred embodiment of the application, the mechanical hand is detachably connected to the wrist joint.

[0015] The technical scheme above enables the mechanical hand to be replaced to complete different operations.

[0016] In a preferred embodiment of the application, the base is a fixed base with no degree of freedom, or the base is a mobile base with a degree of freedom, and the mobile base comprises a mobile platform capable of being remotely controlled, and a connecting base installed on the mobile platform.

[0017] The technical scheme provides two forms of base, i.e., a fixed base and a movable base, and the selection should be made according to actual conditions.

[0018] In another preferred embodiment of the application, a field full-range omnidirectional vision device is installed on each mechanical arm, and the field full-range omnidirectional vision device comprises a plurality of cameras installed on each mechanical arm / working arm to form a 360° surround vision.

[0019] The technical scheme provides a 360° horizontal field of view for each mechanical arm / working arm, and ensures that the mechanical arms and the working arms move without a visual dead angle.

[0020] In another preferred embodiment of the application, the optical axis of the camera is offset from the (n+1)th mechanical arm and is directed toward the joint of the (n+2)th mechanical arm, where n is an integer greater than zero.

[0021] The technical scheme provides a forward field of view for the camera, which passes through the mechanical arms and finally reaches the end working arm, and ensures that the mechanical arms and the working arms move without a visual dead angle.

[0022] In another preferred embodiment of the application, the field working robot further comprises a base column arranged between the base and the multi-stage mechanical arm, the base column comprises a stand column fixedly installed on the base, a sliding seat vertically movable on the stand column, and a telescopic inclined column rotationally connected to the sliding seat through a rotation joint, and the starting end of the first-stage mechanical arm is connected to the end of the telescopic inclined column.

[0023] The technical scheme provides the sliding seat and the rotation joint in the base column, so that the base column has two or more degrees of freedom, which facilitates the multi-stage working arm to extend into the working space from a narrow opening, and improves the practicability of the field working robot.

[0024] In another preferred embodiment of the application, the remote control end comprises a display device and an operation input device, the display device comprises at least one independent display or a head-mounted display for the remote operator to observe the motion of the field device end and the live scene of the surrounding working space, and the operation input device comprises at least one operation platform, and / or an operation handle and / or a remote motion synchronous controller for obtaining the control instruction of the remote operator to the field device end.

[0025] The technical scheme displays the field video transmitted by the field full-range omnidirectional vision device on the display device, the remote operator confirms the surrounding environment of the field working robot through the display device, and the remote operator controls the action of the field working robot through the operation input device according to the surrounding environment of the field working robot.

[0026] In another preferred embodiment of the present application, the image transmission communication link comprises an image transmission link for transmitting image signals of the on-site full-range omnidirectional vision device from the on-site device end to the remote control end, and a data transmission link comprising a forward data transmission link from the on-site device end to the remote control end, and a reverse data transmission link from the remote device end to the on-site control end.

[0027] In another preferred embodiment of the present application, the image transmission communication link can perform non-compressed image transmission with a delay of less than 1 millisecond, or short-delay compressed image transmission with a delay of several milliseconds, or conventional compressed image transmission with a delay of tens of milliseconds.

[0028] In another preferred embodiment of the present application, the non-compressed high-definition video transmission method shown in the following series of patents ZL201580038387.2 "Non-causal prediction signal encoding method and decoding method" and patent ZL201580021704.X "Method for transmitting high-definition video based on transform domain" is used, with a delay of less than 1 millisecond. This high-quality embodiment enables the total delay from the on-site scene to the remote vision to meet the requirements of MR (Mixed Reality) or VR (Virtual Reality) vision, while meeting the extremely low total delay requirements required for high-precision control under on-site device movement or high-speed movement.

[0029] Compared with the prior art, the preferred technical solution of the present application has the following beneficial effects:

[0030] 1) The present application is a super-long, high-degree-of-freedom robot operation system based on remote full-range omnidirectional vision feedback control, which can achieve full-range omnidirectional obstacle avoidance in a complex obstacle workspace, reach a specific operation position and angle of a complex or large workpiece, and replace traditional manual operation with comparable precision and load, achieving safe and efficient unmanned remote operation in complex and hazardous environments.

[0031] 2) By providing on-site full-range omnidirectional vision devices, remote operators can intuitively perceive the on-site scene without personnel on site and modeling, and perform on-site operations through the on-site device end.

[0032] 3) Suitable for remote work in complex obstacle scenarios such as chemical cargo washing / cleaning, dense forest target identification / reconnaissance / search and rescue, etc.

[0033] 4) No scene modeling is required, obstacles are detected by cameras, significantly reducing remote operation delay, and enabling obstacle avoidance while improving control accuracy;

[0034] 5) Cost controllable, compared with remote modeling VR control scheme, no more environmental perception devices and modeling calculation cost.

[0035] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the description or be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0037] Figure 1 is a structural schematic diagram of a work system of an embodiment of the present application used for ship tank cleaning.

[0038] Figure 2 is a schematic diagram of a field full-range omnidirectional vision device in an embodiment of the present application installed on a mechanical arm.

[0039] The reference signs in the drawings accompanying the specification comprise: hatch 100, support column / inclined column 101, pipeline 102, staircase 103, transverse heating pipe 104, base 110, moving platform 111, connecting base 112, base column 120, upright column 121, sliding seat 122, rotating joint 123, telescopic inclined column 124, first-stage mechanical arm 1301, second-stage mechanical arm 1302, third-stage mechanical arm 1303, fourth-stage mechanical arm 1304, fifth-stage mechanical arm 1305, sixth-stage mechanical arm 1306, seventh-stage mechanical arm 1307, eighth-stage mechanical arm 1308, ninth-stage mechanical arm 13019, tenth-stage mechanical arm 13010, eleventh-stage mechanical arm 13011, twelfth-stage mechanical arm 13012, thirteenth-stage mechanical arm 13013, fourteenth-stage mechanical arm 13014, transverse rotating joint 131, extended rear arm 132, longitudinal rotating joint 133, extended front arm 134, base point 135, work arm 140, shoulder joint 141, upper arm 142, elbow joint 143, lower arm 144, wrist joint 145, mechanical hand 146, mechanical hand of high-pressure water gun clamp type 1461, mechanical hand with wiping head 1462, nth-stage mechanical arm 210, nth+1-stage joint 211, nth+1-stage mechanical arm 212, nth+2-stage joint 213, nth+2-stage mechanical arm 214, nth+3-stage joint 215, nth+3-stage mechanical arm 216, omnidirectional vision device 220, first camera 221, second camera 222, third camera 223, fourth camera 224. DETAILED DESCRIPTION

[0040] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application.

[0041] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "vertical", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0042] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication between two elements, it can be direct connection or indirect connection through intermediate medium, and the specific meaning of the above terms can be understood by those skilled in the art according to the specific circumstances.

[0043] The present application provides a high degree of freedom robot operation system based on remote visual feedback control, in a preferred embodiment, the operation system includes a field device end, and a remote control end connected with the field device end through a video communication link, the remote operator can observe the operation state, movement state and surrounding operation space obstacle state of the field device end through remote full-range omnidirectional vision at the remote control end, and feedback control the movement and operation of the field device end.

[0044] Among them, the field device end includes a field operation robot and a field full-range omnidirectional vision device, the field operation robot can adopt a mechanical arm type robot, or a wheeled, foot type, adsorption climbing type mobile robot with a mechanical arm, or an unmanned aerial vehicle type flight robot with a mechanical arm, etc. Remote operation device. The present application mainly takes the field operation robot as a mechanical arm type robot as an embodiment, and other robot modes can be analogized according to the principle of the present application.

[0045] The present application can be used in complex, rugged, closed and multi-obstacle operation space, Figure 1The embodiment shown is used for ship cleaning. At present, ship cleaning mainly relies on manual spraying and wiping of stains. The working environment is poor, and some cabins also store dangerous and toxic substances. To ensure the safety of cleaning personnel and materials, a robot is urgently needed to replace manual labor. The hatch 100 of the working cabin is small, and the internal working space and ground structure of the cabin are very complex. The areas and volumes of each space are divided differently, and various obstacles such as support columns / inclined columns 101, pipelines 102, etc. are crisscrossed, there are stairs 103, and some floors are paved with transverse heating pipes 104 with a diameter of about 20 cm. In addition, there is high-pressure water flow splashing and accumulated water, etc. The robot operation is extremely difficult. Conventional ground mobile robots cannot complete the operation in this complex environment. For example, a design that is too small and light can easily pass through obstacle gaps, but cannot withstand the working back pressure load; a design that can withstand the working back pressure load, but cannot deform, climb, jump over, and drill through obstacles like the human body. Conventional unmanned aerial vehicle type flying robots are also not suitable. Unmanned aerial vehicles with a wingspan of more than one meter can withstand loads of more than a few thousand kilograms. In a complex and blocked cabin space, it is difficult to fly stably and avoid obstacles, and it is easy to collide and cause equipment damage.

[0046] The field operation robot of the embodiment adopts a fixedly installed mechanical arm type robot. To increase the load capacity and motion rigidity of the end of the mechanical arm (to avoid elastic shaking caused by the reaction force during operation and reduce the operation precision), the field operation robot in the embodiment is a three-body composite structure type mechanical arm.

[0047] Specifically, the field operation robot includes a base 110 installed on the hatch 100, a base column 120 installed on the base 110, a multi-stage mechanical arm installed at the end of the base column 120, and an operation arm 140 installed at the end of the last stage mechanical arm. The end of the last stage mechanical arm is called a base point 135, and the operation arm 140 is installed on the base point 135. Figure 1 The fourteen-stage mechanical arm is shown. From the base column 120 side to the operation arm 140 side, they are the first-stage mechanical arm 1301,..., and the fourteenth-stage mechanical arm 13014 in order. The fourteenth-stage mechanical arm 13014 is the last stage mechanical arm. A field full-range omnidirectional vision device is installed at the joint between the operation arm 140 and each stage mechanical arm. The field full-range omnidirectional vision device provides the required omnidirectional field of view for this stage mechanical arm.

[0048] The base 110 is a high-precision rigid body with great load, which can withstand great torque without displacement or deformation. In an embodiment, the base 110 is a fixed base with no degree of freedom, which is fixedly installed on the hatch 100. In another embodiment, the base 110 is a movable base with degrees of freedom, which includes a movable platform 111 capable of remote control and a connecting base 112 installed on the movable platform 111. The movable platform 111 includes a movable vehicle platform, a movable robot platform, or a drone platform, etc. The movable base has more than 1 to 3 degrees of freedom. It should be noted that when the size of the hatch 100 is large enough for the on-site operation robot to enter directly, the base 110 can be installed at the bottom of the cabin.

[0049] The base column 120 is a high-precision rigid body with great load, which can withstand great torque with little displacement and deformation. In the present application, the base column 120 includes a column 121 fixedly installed on the base 110, a sliding seat 122 vertically movable on the side of the column 121, and an extension inclined column 124 rotatably connected with the sliding seat 122 through a rotary joint 123. The base column 120 has two or more degrees of freedom (at least including one moving pair and one rotating pair). In the present embodiment, the extension inclined column 124 is composed of two or more telescopic mechanical arms, which have various structures such as sleeve type and track sliding type, etc. and are prior art, which will not be described in detail. At the beginning of the operation, by controlling the vertical movement of the sliding seat 122 and the action of the rotary joint 123, the extension inclined column 124 can be completely extended into the cabin from outside the cabin through the hatch 100, avoiding the stairs 103. After the operation is completed, it is completely retracted out of the cabin.

[0050] The structure of each level of the mechanical arm is the same, and the size of each level of the mechanical arm can be the same or different. The first level of the mechanical arm 1301 is taken as an example for description. Each level of the mechanical arm includes a transverse rotary joint 131, an extension rear arm 132, a longitudinal rotary joint 133, and an extension front arm 134. The transverse rotary joint 131 of the next level of the mechanical arm is connected with the extension front arm 134 of the previous level of the mechanical arm, and the extension rear arm 132 of the first level of the mechanical arm 1301 is rotatably connected with the end of the extension inclined column 124 through the transverse rotary joint 131. By controlling the longitudinal rotary joint 133, the extension rear arm 132, and the extension front arm 134, the mechanical arm can be straightened or folded longitudinally. In addition, by controlling the transverse rotary joint 131, the multi-level mechanical arm can be twisted and "snaked" in the three-dimensional operation space, so as to realize multi-plane twisted traversal in the complex obstacle space, thereby realizing obstacle avoidance in the complex three-dimensional obstacle space. At the same time, after the mechanical arm is folded in a zigzag shape, the super-long multi-level mechanical arm can be shrunk to a smaller size, so that it can be extended into the cabin through the hatch 100 level by level to carry out the cabin cleaning work.

[0051] In the present application, the work arm 140 adopts a structure similar to the human arm, including an upper arm 142 rotatably connected to the last stage mechanical arm 13014 through a shoulder joint 141, a lower arm 144 rotatably connected to the upper arm through an elbow joint 143, and a mechanical hand 146 rotatably connected to the lower arm 144 through a wrist joint 145. The mechanical hand 146 is detachably connected to the wrist joint 145, so that the mechanical hand 146 can be replaced. In the high-pressure water washing operation, a high-pressure water gun clamp type mechanical hand 1461 is used, and in the wiping operation, a mechanical hand 1462 with a wiping head is used, etc. The work arm 140 adopts a structure similar to the human arm, so that a human body sensing device or a human body motion visual recognition device can be used at the remote control end. The remote operator can control the work arm 140 on-site operation through the movement of his arm, which is intuitive, visual and efficient.

[0052] In Figure 1 In the complex, rugged, closed and multi-obstacle operation space shown, the movement of the fourth stage mechanical arm 1304, the eighth mechanical arm 1308, and the fourteenth stage mechanical arm 13014 from the base column 120 to the mechanical hand 146 at the end of the work arm 140 is limited by the inclined column 101 and the pipeline 102 of the movement plane, and cannot directly reach. By bending the fourth stage mechanical arm 1304 to avoid the obstacle of the inclined column 101, bending the eighth mechanical arm 1308 to avoid the obstacle of the pipeline 102, and bending the fourteenth stage mechanical arm 13014 to bypass the obstacle of the pipeline 102, the multi-plane "snake" type twisting is finally reached in the complex obstacle space to reach the cleaning operation position, and then the cleaning operation is performed through the work arm 140. Through the present application, the cabin cleaning task in the complex obstacle scene is solved by using a super-long, multi-joint high-degree-of-freedom advanced mechanical arm.

[0053] The on-site full-range omnidirectional vision device of the present application is a multi-stage omnidirectional imaging system that can be seen in the complex obstacle space on site, through the obstacles, so that each joint of the multi-stage mechanical arm can be seen in the 360° range in the horizontal direction. After visual processing, the Figure 1 In the complex obstacle space shown, a visible channel is formed from the base 110 to the mechanical hand 146 at the end of the work arm 140, which has a longitudinal full-range field of view without dead angle and a horizontal 360° without dead angle, so that the movement position of each stage mechanical arm in the multi-obstacle operation space around it is in the field of view, thereby enabling the remote operator to have full-range omnidirectional obstacle avoidance observation capability.

[0054] Specifically, as Figure 2 As shown, the on-site full-range omnidirectional vision device is installed at each joint of the mechanical arm, which can sense the cabin in real time. For the sake of simplicity, only the three-stage omnidirectional vision device for the four-stage mechanical arm is shown in this embodiment, and the omnidirectional vision device on the remaining stage mechanical arm is similar. Figure 2The shown four-stage mechanical arm is the nth-stage mechanical arm 210 (n is an integer greater than or equal to 0), the (n+1)th-stage mechanical arm 212 connected to the end of the nth-stage mechanical arm 210 through the (n+1)th joint 211, the (n+2)th-stage mechanical arm 214 connected to the end of the (n+1)th-stage mechanical arm 212 through the (n+2)th joint 213, and the (n+3)th-stage mechanical arm 216 connected to the end of the (n+2)th-stage mechanical arm 214 through the (n+3)th joint 215. The (n+1)th joint 211, the (n+2)th joint 213, and the (n+3)th joint 215 are all rotary joints with two degrees of freedom, i.e., they can rotate a certain angle in the horizontal left-right and vertical up-down directions at the same time.

[0055] The required on-site full-range omnidirectional vision device 220 of the (n+1)th-stage mechanical arm 212 includes multiple cameras, such as four cameras 221, 222, 223, and 224, which constitute 360° surround vision. The four cameras are symmetrically or nearly symmetrically installed on the four sides of the cross section at the start of the (n+1)th-stage mechanical arm 212, the horizontal field of view angle of each camera is 180 degrees, and the camera optical axis is offset from the (n+1)th-stage mechanical arm 212 and faces the (n+2)th joint 213, which meets the requirement of 360° horizontal surround vision stitching. After the surround vision stitching processing, the four cameras constitute 360° surround vision, which provides the (n+1)th-stage mechanical arm 212 with a horizontal 360° non-blind forward view.

[0056] Similarly, the four cameras of the required on-site full-range omnidirectional vision device 230 of the (n+2)th-stage mechanical arm 214 are installed on the four sides of the cross section at the start of the (n+2)th-stage mechanical arm 214 to ensure the required forward view during the movement of the (n+2)th-stage mechanical arm 214. Similarly, the non-blind omnidirectional view required by the (n+3)th-stage mechanical arm 216 is provided by the on-site full-range omnidirectional vision device 240. This camera gradually provides a forward view from the base 110 through each stage of the mechanical arm to the end manipulator 146, ensuring that each stage of the mechanical arm moves without a visual dead angle and solving the problem of visual perception in a complex obstacle environment for remote operation. Preferably, the 360° horizontal surround vision is 360° horizontal stereo surround vision, which is more conducive to the operator's perception of the size and distance of the obstacles on site.

[0057] It should be noted that the on-site full-range omnidirectional vision device does not need to be equipped with a camera at the joint of each level of the mechanical arm. For example, cameras are installed at some positions in the scene and on the mechanical arm, which can monitor the movable positions of the multi-level mechanical arm in a control cycle, without the need to install a camera for each level of the mechanical arm. For another example, when the joint of the mechanical arm is limited to movement within a fixed angle, the obstacles outside the moving angle of the mechanical arm do not need to be observed, thereby reducing the monitoring of the cameras for these angles that do not need to be observed. For another example, when the length of the mechanical arm is too long, too wide or has an angle, causing the movement of the mechanical arm to have a blind area, a camera is additionally added in the visual blind area of the mechanical arm according to the need. In addition, the on-site full-range omnidirectional vision device can replace the camera according to the actual scene, such as replacing it with a night-vision infrared camera in a dark scene, and can also carry other auxiliary modules to assist the remote operator in perceiving the scene. For another example, a sound module is added to capture on-site audio data.

[0058] In the present application, the remote control end includes a display device and an operation input device. The display device includes at least one independent display or a head-mounted display, which is used for the remote operator to observe the movement of the on-site device end and the surrounding working space scene live. The on-site picture displayed by the display device can use a camera switching mode, can be spliced into a simultaneous display mode of multiple cameras, can add a recognition system on the captured picture, and can use artificial intelligence control based on the above. The operation input device includes at least one operation platform (such as a touch screen, a keyboard, a mouse), and / or an operation handle and / or a remote motion synchronous controller, which are used to obtain the control instructions of the remote operator to the on-site device end. It is preferred to simultaneously set the operation platform, the operation handle and the remote motion synchronous controller. The operation input device in the embodiment of a specific scene is an operator with a display screen, which combines the on-site working robot and the on-site full-range omnidirectional vision device on one machine.

[0059] In the present application, the image transmission link includes an image transmission link and a data transmission link, and the medium of the image transmission link includes transmission modes including optical cable, network cable and wireless image transmission.

[0060] The image transmission link is used to transmit the image signal of the full-range and full-direction vision of the application from the field device end to the remote control end. In order to enable the remote operator to accurately observe and perceive the fast-moving mechanical arm in the field, the image transmission link needs to use a non-delayed or extremely low-delay image transmission method. In a high-quality embodiment, the image transmission link adopts the non-compressed high-definition video transmission method of the following series of patents PCT / CN2015 / 084897 "Non-causal video prediction encoding method in non-causal prediction image and video encoding and decoding method" and patent application PCT / CN2015 / 080054 "Method for transmitting high-definition video based on transform domain", the image transmission delay is below 1ms, and the high-quality embodiment enables the total delay from the field scene to the remote vision to meet the requirements of MR (mixed reality) or VR (virtual reality) vision, while meeting the requirement of extremely low total delay required by high-precision control under the condition of high-speed movement in the field; in another reduced-quality embodiment, the image transmission link adopts a low-delay image transmission compression image transmission with only video intra prediction, and the image transmission delay is several milliseconds; in still another embodiment with lower quality, the image transmission link adopts a compressed image transmission with video inter prediction, and the image transmission delay is tens of milliseconds. The aforementioned image transmission delay refers to all transmission and processing delays between the output video signal of the camera at the field device end and the input video signal of the display device at the remote end, which generally includes image transmission device delay, air or wired propagation delay, and image transmission device delay.

[0061] The data transmission link is a bidirectional data transmission link between the field device end and the remote control end, and includes a forward data transmission link from the field device end to the remote control end and a reverse data transmission link from the remote device end to the field control end. The forward data transmission link is used to transmit the state information of the field device end to the remote device end, and the reverse data transmission link is used to transmit the control information of the remote device end to the field device end.

[0062] In the preferred embodiment, the working steps of the operation system of the application are as follows:

[0063] 1) The field personnel move the field operation robot to the vicinity of the hatch 100 through the base 110, and the remote operator confirms the environment around the field operation robot through the display device of the remote control end;

[0064] 2) Starting from the first-stage mechanical arm 1301 of the field operation robot close to the base 110, the remote operator controls the corresponding mechanical arm action through the operation input device according to the camera picture of each-stage mechanical arm;

[0065] 3) Remote operator controls the mechanical arm action step by step according to the live video transmitted by the live all-around visual equipment through the data transmission link until the live operation robot and the mechanical hand 146 are moved to the position to be cleaned in the cabin;

[0066] 4) Remote operator controls the end mechanical hand 146 to unfold and clean according to the live video transmitted by the live all-around visual equipment through the remote body sense synchronous controller of the input device of the remote control terminal;

[0067] 5) The live operation robot continues to operate, observes the position to be cleaned in the cabin through the camera data of each level of mechanical arm, and cleans until the cleaning work is completed.

[0068] In the description of the present specification, the description referring to the terms "preferred embodiment", "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0069] Although the embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A high-degree-of-freedom robot operation system based on remote visual feedback control, characterized in that, It includes a field device terminal and a remote control terminal connected to the field device terminal via image transmission communication. The remote operator can remotely observe the operation status, movement status, and obstacle status of the surrounding work space of the field device terminal through the remote control terminal, and provide feedback to control the movement and operation of the field device terminal. The field equipment includes a field operation robot and a field omnidirectional vision device. The field operation robot includes a base, a multi-stage robotic arm mounted on the base, and a working arm mounted at the end of the last stage robotic arm. The field omnidirectional vision device is installed on the working arm and each or some stages of the robotic arm, and provides the required omnidirectional field of view to the working arm and the robotic arm. The on-site omnidirectional vision device is installed on each level of the robotic arm. The on-site omnidirectional vision device includes multiple cameras installed at the beginning of each level of robotic arm / working arm, and the multiple cameras form a 360° surround vision. The camera's optical axis deflects outward from the (n+1)th level robotic arm and toward the joint of the (n+2)th level robotic arm, where n is an integer greater than zero.

2. The high-degree-of-freedom robot operation system based on remote visual feedback control according to claim 1, characterized in that, Each stage of the robotic arm includes a transverse rotation joint, an extended rear arm, a longitudinal rotation joint, and an extended forearm connected in sequence. Each stage of the robotic arm can be extended or folded longitudinally through the longitudinal rotation joint, the extended rear arm, and the extended forearm. At the same time, the transverse rotation joint enables the multi-stage robotic arm to twist and "snake" segment by segment in the three-dimensional working space.

3. The high-degree-of-freedom robot operation system based on remote visual feedback control according to claim 1, characterized in that, The working arm includes an upper arm connected to the end of the last stage robotic arm via a shoulder joint, a lower arm connected to the upper arm via an elbow joint, and a robotic hand connected to the lower arm via a wrist joint.

4. A high-degree-of-freedom robot operation system based on remote visual feedback control according to claim 3, characterized in that, The robotic arm is detachably connected to the wrist joint.

5. A high-degree-of-freedom robot operation system based on remote visual feedback control according to claim 1, characterized in that, The base is a fixed base with no degrees of freedom, or the base is a mobile base with degrees of freedom. The mobile base includes a remotely controllable mobile platform and a connecting base installed on the mobile platform.

6. A high-degree-of-freedom robot operation system based on remote visual feedback control according to any one of claims 1-5, characterized in that, The on-site operation robot also includes a base column located between the base and the multi-stage robotic arm. The base column includes a column fixedly installed on the base, a sliding seat that can move vertically on the column, and a telescopic inclined column that is rotatably connected to the sliding seat through a rotating joint. The beginning of the first-stage robotic arm is connected to the end of the telescopic inclined column.

7. A high-degree-of-freedom robot operation system based on remote visual feedback control according to any one of claims 1-5, characterized in that, The remote control terminal includes a display device and an operation input device. The display device includes at least one independent display or head-mounted display for remote operators to observe the movement of the field equipment and the real-time situation of the surrounding work space. The operation input device includes at least one operating platform, and / or operating handle and / or remote motion-sensing synchronization controller for obtaining control commands from remote operators to the field equipment.

8. A high-degree-of-freedom robot operation system based on remote visual feedback control according to any one of claims 1-5, characterized in that, The image transmission communication link includes an image transmission link and a data transmission link. The image transmission link is used to transmit image signals from the field device to the remote control terminal. The data transmission link includes a forward data transmission link from the field device to the remote control terminal and a reverse data transmission link from the remote device to the field control terminal.

9. A high-degree-of-freedom robot operation system based on remote visual feedback control according to claim 8, characterized in that, The image transmission communication link can perform uncompressed image transmission with a delay of less than 1 millisecond, or short-delay compressed image transmission with a delay of several milliseconds, or conventional compressed image transmission with a delay of tens of milliseconds.

Images