Real-time interactive working system and method of stereo vision for master-slave remote operating robot in distribution network

Abstract

The application belongs to the technical field of distribution network master-slave remote operation robots, and provides a kind of distribution network master-slave remote operation robot stereo vision real-time interactive operation system and method. Wherein, the system includes high-altitude monitoring system and ground monitoring and VR display interactive system in mutual communication;The ground monitoring and VR display interactive system includes ground monitoring module and VR display interactive equipment;VR display interactive equipment is used to form corresponding control instruction through voice control instruction or operator movement, to control high-altitude vision module, tool vision module and ground monitoring module to carry out visual adjustment or switching;Splice fusion received multi-channel video information, form virtual scene based on digital twin;In the virtual scene based on digital twin, the mapping relationship of distribution network line, robot, operation tool and actual operation working condition is constructed, so as to complete operation task in virtual scene and realize remote operation.

Description

Technical Field

[0001] This invention belongs to the field of distribution network master-slave teleoperation robot technology, and particularly relates to a stereo vision real-time interactive operation system and method for distribution network master-slave teleoperation robot. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Master-slave teleoperated robots are mainly used for live-line work on power distribution networks, performing tasks such as disconnecting live leads, replacing surge arresters and drop-out switches, etc., completely isolating workers from the high-voltage electric field and maximizing their safety. Currently, live-line working robots are operated remotely by operators on the ground, with the robotic arm rising to the work site along with an aerial work platform to perform live-line work.

[0004] The inventors discovered that operators cannot observe the status of the end-effector tools at close range, making it difficult to grasp the relative positional relationship between the end-effector and the target. Because single-view vision devices cannot present information such as whether the tools are in place, the on-site information obtained by operators through cameras lacks three-dimensional information. In particular, when the robotic arm is operating, there is also some obstruction, making it difficult for operators to accurately obtain the positional relationship of the target on the screen, which poses a great challenge to remote operation and results in low work efficiency. Summary of the Invention

[0005] To address the technical problems mentioned above, this invention provides a real-time interactive operation system and method for a master-slave teleoperation robot in a power distribution network, which enables operators to accurately obtain the positional relationships of objects at the operation site, and to perform comprehensive three-dimensional monitoring of high-altitude operations from the ground, thereby improving teleoperation efficiency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides a real-time interactive operation system for a master-slave teleoperated robot in a power distribution network, comprising a high-altitude monitoring system and a ground monitoring and VR display interactive system that communicate with each other.

[0008] The high-altitude monitoring system includes a high-altitude vision module and a tool vision module. The high-altitude vision module and the tool vision module are used to detect video information of the high-altitude operation object, video information of the arm carrying the operation tool, and video information of the operation tool status, respectively, and transmit them to the ground monitoring and VR display interaction system.

[0009] The ground monitoring and VR display interaction system includes a ground monitoring module and a VR display interaction device. The ground monitoring module is used to detect the ground status video information of the high-altitude operation equipment and transmit it to the VR display interaction device.

[0010] The VR display and interaction device is used for:

[0011] The corresponding control commands are generated through voice control commands or operator movements to control the aerial vision module, tool vision module and ground monitoring module to make visual adjustments or switches.

[0012] The received video information is spliced ​​and fused to form a virtual scene based on digital twin;

[0013] In a digital twin-based virtual scenario, a mapping relationship is constructed between power distribution lines, robots, work tools, and actual working conditions to complete work tasks in the virtual scenario and realize remote operation.

[0014] In one implementation, the ground monitoring and VR display interaction system also communicates with a remote monitoring center via 5G, which is used to monitor the entire process of high-altitude operations.

[0015] In one implementation, the aerial vision module includes a platform camera, a platform dome camera, and a wrist camera; the platform camera is used to observe the overall information of the aerial work platform; the platform dome camera is used to observe the bottom information of the work platform and the replacement information of the work tools in real time; and the wrist camera is used to detect the movement status of the arm carrying the work tools.

[0016] In one implementation, the high-altitude vision module includes a binocular camera; the binocular camera is used to simulate the human eye's state when observing the work content.

[0017] In one implementation, a binocular camera gimbal is provided at the bottom of the binocular camera. The VR display interaction device detects the operator's neck rotation and synchronously controls the movement of the binocular camera gimbal to achieve rotation of the binocular camera's field of view, thereby enabling omnidirectional acquisition of video images.

[0018] In one implementation, the ground monitoring module includes an insulated boom truck vision submodule and an on-board control room vision submodule. The insulated boom truck vision submodule is used to monitor the position and attitude information of the insulated boom during operation. The on-board control room vision submodule is used to observe the information displayed in the on-board control room and assist the operator in working in the on-board control room.

[0019] As one implementation, the high-altitude monitoring system also includes an environmental information monitoring module, which is used to detect environmental information around the high-altitude work object and transmit it to the ground monitoring and VR display interaction system.

[0020] As one implementation, the environmental information monitoring module includes a five-element meteorological instrument, a platform tilt sensor, and a leakage current sensor, used to detect meteorological information, the horizontal and pitch angles of the aerial work platform, and the leakage current of the aerial work equipment, respectively.

[0021] As one implementation method, the VR display interaction device has two operation modes: manual touch and voice control. The manual touch mode is used by the operator to start and debug the system, while the voice control mode is used by the operator to switch the vision system and adjust the viewing angle during operation.

[0022] As one implementation method, during the process of stitching and fusing the received multi-channel video information, the stitching and fusing of multi-channel video is achieved based on the correspondence and matching relationship between the robot and the work object related features between consecutive image frames of the multi-channel video information.

[0023] As one implementation, the characteristics related to the work object include the position, shape, texture, and color of the work object.

[0024] A second aspect of the present invention provides an interaction method based on the stereo vision real-time interactive operation system of the master-slave teleoperated robot in a power distribution network as described above, comprising:

[0025] VR display and interaction devices generate corresponding control commands through voice control commands or operator movements to control the high-altitude vision module, tool vision module, and ground monitoring module to adjust or switch vision.

[0026] VR display and interaction devices stitch together and merge video information received from high-altitude vision modules, tool vision modules, and ground monitoring modules to form a virtual scene based on digital twins. In the virtual scene based on digital twins, a mapping relationship between power distribution lines, robots, work tools, and actual working conditions is constructed to complete work tasks in the virtual scene and realize remote operation.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] (1) An innovative three-dimensional vision real-time interactive operation technology for distribution network master-slave teleoperation robot was proposed, and a three-dimensional vision real-time interactive operation system for distribution network master-slave teleoperation robot was developed. A corresponding virtual scene was formed by using VR display interaction equipment. The high-altitude vision module, tool vision module and ground monitoring module were adjusted or switched by voice control commands or corresponding control commands formed by the operator's movement. This solved the problem of low efficiency of teleoperation high-altitude operation, enabled the operator to accurately obtain the position of the operation object on the operation site, and improved the efficiency of teleoperation.

[0029] (2) An innovative multi-channel stereoscopic video fusion and digital twin virtual scene technology is proposed. It forms a virtual scene based on digital twin by splicing and fusing the received multi-channel video information. In the virtual scene based on digital twin, the mapping relationship between the power distribution line, robot, operation tool and actual operation condition is constructed so as to complete the operation task in the virtual scene. This solves the problem of low efficiency of remote operation of high-altitude operation, realizes remote operation, and improves the efficiency of remote operation.

[0030] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0032] Figure 1 This is a schematic diagram of the real-time interactive operation system for a master-slave teleoperated robot in a power distribution network according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the structure of the distribution network master-slave teleoperation robot stereo vision real-time interactive operation system according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the high-altitude vision module structure according to an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the environmental information monitoring module structure according to an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of a ground monitoring and VR display interaction system according to an embodiment of the present invention;

[0037] Figure 6 This is a schematic diagram of the VR display interaction module according to an embodiment of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] Example 1

[0042] Reference Figure 1 and Figure 2 This embodiment provides a real-time interactive operation system for a distribution network master-slave teleoperated robot with stereo vision, which includes a high-altitude monitoring system 1 and a ground monitoring and VR display interactive system 2 that communicate with each other.

[0043] In the actual implementation process, the high-altitude monitoring system 1 and the ground monitoring and VR display interaction system 2 transmit data through optical fiber.

[0044] In some specific embodiments, the ground monitoring and VR display interaction system 2 also communicates with a remote monitoring center via 5G, and the remote monitoring center is used to monitor the entire process of high-altitude operations.

[0045] The high-altitude monitoring system 1 in this embodiment includes a high-altitude vision module and a tool vision module, and includes a high-altitude monitoring system and a ground monitoring and VR display interaction system that communicate with each other; the high-altitude vision module and the tool vision module are used to detect video information of the high-altitude operation object, video information of the arm carrying the operation tool, and video information of the operation tool status, respectively, and transmit them to the ground monitoring and VR display interaction system.

[0046] Among them, such as Figure 3 As shown, the high-altitude vision module includes a platform bullet camera 3, a platform dome camera 7, and wrist cameras (left wrist camera 4 and right wrist camera 5);

[0047] The platform gun 3 is used to observe the overall information of the aerial work platform; for example, the platform gun 3 is installed on the platform control box and is equipped with an electric lifting device 10, which can realize three control methods: platform point control, controller remote control and control room touch control.

[0048] The platform hemispherical camera 7 is used to monitor the bottom information of the work platform and the replacement information of the work tools in real time; for example, it is installed on the lower front side of the work platform.

[0049] Wrist cameras are used to detect the movement of the slave arm carrying the work tool. For example, a robotic arm hand-eye camera can be installed at the wrist of the robotic arm (slave arm) for macro monitoring.

[0050] In the specific implementation process, the high-altitude vision module includes a binocular camera 6; the binocular camera 6 is used to simulate the human eye state to observe the operation content.

[0051] The binocular camera 6 is equipped with a binocular camera gimbal at its bottom. The VR display interaction device detects the operator's neck rotation and synchronously controls the movement of the binocular camera gimbal to achieve rotation of the binocular camera's field of view, thereby enabling omnidirectional acquisition of video images.

[0052] For example, the binocular camera 6 is installed at the front of the work platform and can automatically adjust the video brightness in real time according to the ambient light. Its omnidirectional gimbal system can achieve two degrees of freedom of pitch and rotation. The binocular camera has the characteristics of sensing the depth information of the work and can simulate the state of the human eye to observe the work content.

[0053] In some embodiments, the high-altitude monitoring system further includes an environmental information monitoring module, which is used to detect environmental information around the high-altitude work object and transmit it to the ground monitoring and VR display interaction system.

[0054] Specifically, such as Figure 4 As shown, the environmental information monitoring module includes a five-element meteorological instrument 8, a platform tilt sensor 9, and a leakage current sensor, which are used to detect meteorological information, the horizontal and pitch angles of the aerial work platform, and the leakage current of the aerial work equipment, respectively.

[0055] For example, the five-element weather instrument 8 is installed on the platform control box and can collect environmental information such as wind speed, wind direction, temperature, humidity and air pressure;

[0056] The platform tilt sensor 9 is installed on the platform control box to detect the horizontal and pitch angles of the aerial work platform.

[0057] The leakage current sensor is installed at the end of the insulated boom truck to detect the leakage current of the entire vehicle and transmits data with the aerial work platform router via Wi-Fi.

[0058] In the specific implementation process, the tool vision module includes tool vision such as electric peelers and connectors. The working tools are equipped with cameras, which use the working tool power system and transmit data via Wi-Fi. The router located on the aerial work platform receives the video data and the front-end industrial control computer processes the video data.

[0059] The visual and data transmission between the high-altitude monitoring system and the ground monitoring and VR display interaction system is mainly in the form of Wi-Fi + fiber optic transmission. The visual signals of the tools are transmitted via Wi-Fi and received by a router installed on the high-altitude work platform. The front-end industrial control computer and the router in the high-altitude monitoring system are connected to a switch, and the switch is connected to a fiber optic converter. The video and control information is transmitted to the ground monitoring and VR display interaction system via fiber optic cable.

[0060] Reference Figure 5 and Figure 6 The ground monitoring and VR display interaction system includes a ground monitoring module and a VR display interaction device. The ground monitoring module is used to detect ground status video information of the high-altitude work equipment and transmit it to the VR display interaction device. The VR display interaction device is used for:

[0061] The corresponding control commands are generated through voice control commands or operator movements to control the aerial vision module, tool vision module and ground monitoring module to make visual adjustments or switches.

[0062] The received video information is spliced ​​and fused to form a virtual scene based on digital twin;

[0063] In a digital twin-based virtual scenario, a mapping relationship is constructed between power distribution lines, robots, work tools, and actual working conditions to complete work tasks in the virtual scenario and realize remote operation.

[0064] In the process of stitching and fusing the received multi-channel video information, the stitching and fusion of multi-channel video is achieved based on the correspondence and matching relationship between the robot and the operation object related features between the continuous image frames of the multi-channel video information.

[0065] The relevant features of the work object include the position, shape, texture, and color of the work object.

[0066] During the stitching and fusion of multiple video feeds, a model inference acceleration algorithm is used to achieve millisecond-level latency-free detection of tasks within the video images. This model inference acceleration algorithm is an existing algorithm and will not be detailed here.

[0067] The ground monitoring module includes an insulated boom truck vision submodule and an onboard control room vision submodule. The insulated boom truck vision submodule monitors the position and attitude information of the insulated boom during operation. The onboard control room vision submodule observes the information displayed in the onboard control room and assists operators in their work within the control room. The insulated boom truck vision submodule consists of a roof-mounted camera 12 installed on the upper side of the insulated boom truck control room and a boom-supported camera 11, capable of displaying multi-angle video of the surrounding space. The onboard control room vision submodule consists of a left onboard camera 13 and a right onboard camera 14.

[0068] In practice, VR display interaction devices have two operation modes: manual touch and voice control. The manual touch mode is used by operators to start and debug the system, while the voice control mode is used by operators to switch the vision system and adjust the viewing angle during operation.

[0069] In some embodiments, the VR display interaction device includes a VR display module and a display interaction module.

[0070] In this embodiment, the VR display module takes a VR display headset as an example:

[0071] The VR display headset comes with a built-in microphone and headphones. Operators can interact with the monitoring system via voice through the microphone and receive voice feedback from the display interaction module through the headphones. The visual sub-module in the vehicle control room allows for direct observation and operation of the situation within the control room. The display interaction module wirelessly pushes video over the 5G public network via a 5G module. The public network stream is sent to the control center, where it decodes and displays the video, providing a clear view of the operators and their work, enabling real-time scheduling and guidance.

[0072] Operators can issue control commands for each camera via the microphone on the VR headset. The voice control system is deployed on the vehicle-mounted industrial computer. This system can automatically recognize the operator's voice commands, convert the voice information into control instructions through semantic analysis, and issue them to various devices for related display and camera parameter adjustments. Voice control can also control the button content displayed on all screens 15. When adjusting the camera lens angle and focus, camera switching is required first.

[0073] VR devices can detect the operator's neck rotation and synchronously control the gimbal movement of the binocular camera. By rotating the VR display helmet, the camera's field of view can be rotated, and the content displayed inside the helmet is refreshed in real time, giving the operator a better sense of being on-site.

[0074] Table 1. Voice Control Command Contents

[0075]

[0076] Among them, the display screen 15 is the structure of the display function in the display interaction module.

[0077] The display screen 15 consists of a left screen, a middle screen, and a right screen, arranged in a wraparound shape with adjustable angles. The displayed content, including but not limited to the video display schemes shown in the remote monitoring center, can be optimized and adjusted according to on-site needs. The middle screen is the main display screen, and its main interface includes video display, instrument parameters, tool parameters, and system settings. Video display buttons allow switching between the left and middle screens and individual videos. The left and middle screens primarily display visual information from the high-altitude monitoring system and the ground monitoring system, accurately obtaining the positional relationships and work content of objects at the operation site through multiple vision devices, assisting operators in completing their tasks.

[0078] The VR display helmet defaults to displaying content captured by the binocular cameras in the high-altitude monitoring system. Rotating the VR display helmet rotates the camera's viewpoint, and the content displayed inside the helmet updates in real time. Video channel switching latency is ≤1 second, providing operators with a better sense of being on-site.

[0079] The VR headset can switch between displaying multiple video feeds (visual display content), including video from high-altitude and ground monitoring systems. The headset's binocular cameras can switch between video from the onboard control room cameras, allowing observation of environmental information and live-line work tool information on the data display screen. It also monitors the status and operation of the onboard control room, facilitating human-machine interaction.

[0080] Voice control is an important component of the vision system of live-line working robots. In actual operation, the operator's attention is mainly focused on controlling the robotic arm. If the user needs to manually adjust parameters such as the position of the camera pan-tilt unit while controlling the robotic arm, it will not only affect the work efficiency, but also reduce the safety of the operation due to the distraction.

[0081] Operators can issue control commands for each camera via the microphone on the VR headset. The voice control system is deployed on the vehicle-mounted industrial computer. This system can automatically recognize the operator's voice commands, convert the voice information into control instructions through semantic analysis, and issue them to various devices for related display and camera parameter adjustments. Voice control can also control the button content displayed on all monitors. When adjusting the camera lens angle and focus, camera switching is required first.

[0082] Based on 5G communication technology, the vehicle-mounted industrial control computer pushes all video streams and environmental data information to the remote monitoring center via the 5G network, enabling the remote monitoring center to monitor the entire operation process.

[0083] It should be noted that the VR display module can also be implemented using other existing VR display and interaction devices, which will not be discussed in detail here.

[0084] Example 2

[0085] This embodiment provides an interaction method based on the real-time interactive operation system of a master-slave teleoperated robot in a power distribution network as described above, which includes:

[0086] VR display and interaction devices generate corresponding control commands through voice control commands or operator movements to control the high-altitude vision module, tool vision module, and ground monitoring module to adjust or switch vision.

[0087] VR display and interaction devices stitch together and merge video information received from high-altitude vision modules, tool vision modules, and ground monitoring modules to form a virtual scene based on digital twins. In the virtual scene based on digital twins, a mapping relationship between power distribution lines, robots, work tools, and actual working conditions is constructed to complete work tasks in the virtual scene and realize remote operation.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time interactive operation system for a master-slave teleoperated robot in a power distribution network, characterized in that, This includes interconnected high-altitude monitoring systems and ground-based monitoring and VR display interactive systems; The high-altitude monitoring system includes a high-altitude vision module and a tool vision module. The high-altitude vision module and the tool vision module are used to detect video information of the high-altitude operation object, video information of the arm carrying the operation tool, and video information of the operation tool status, respectively, and transmit them to the ground monitoring and VR display interaction system. The high-altitude vision module includes a platform camera, a platform dome camera, and a wrist camera; the platform camera is used to observe the overall information of the high-altitude work platform; the platform dome camera is used to observe the bottom information of the work platform and the replacement information of the work tools in real time; the wrist camera is used to detect the movement status of the arm carrying the work tools; the high-altitude vision module includes a binocular camera; the binocular camera is used to simulate the human eye to observe the work content; The ground monitoring and VR display interaction system includes a ground monitoring module and a VR display interaction device. The ground monitoring module is used to detect the ground status video information of the high-altitude operation equipment and transmit it to the VR display interaction device. The VR display and interaction device is used for: The corresponding control commands are generated through voice control commands or operator movements to control the aerial vision module, tool vision module and ground monitoring module to make visual adjustments or switches. The stitching and fusion of multiple video information received from the high-altitude vision module, tool vision module, and ground monitoring module forms a virtual scene based on digital twin. During the stitching and fusion of the multiple video information received, the stitching and fusion of multiple videos is achieved based on the corresponding matching relationship between the robot and the work object related features between the continuous image frames of the multiple video information. In a digital twin-based virtual scenario, a mapping relationship is constructed between power distribution lines, robots, work tools, and actual working conditions to complete work tasks in the virtual scenario and realize remote operation.

2. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 1, characterized in that, The ground monitoring and VR display interaction system also communicates with the remote monitoring center via 5G, which is used to monitor the entire process of high-altitude operations.

3. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 1, characterized in that, The binocular camera is equipped with a binocular camera gimbal at its bottom. The VR display interaction device detects the operator's neck rotation and synchronously controls the movement of the binocular camera gimbal to achieve rotation of the binocular camera's field of view, thereby enabling omnidirectional acquisition of video images.

4. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 1, characterized in that, The ground monitoring module includes an insulated bucket truck vision submodule and an on-board control room vision submodule. The insulated bucket truck vision submodule is used to monitor the position and attitude information of the insulated lifting arm during operation. The vehicle control room vision submodule is used to observe the information displayed in the vehicle control room and assist operators in their work in the vehicle control room.

5. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 1, characterized in that, The high-altitude monitoring system also includes an environmental information monitoring module, which is used to detect environmental information around the high-altitude work object and transmit it to the ground monitoring and VR display interaction system.

6. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 5, characterized in that, The environmental information monitoring module includes a five-element meteorological instrument, a platform tilt sensor, and a leakage current sensor, which are used to detect meteorological information, the horizontal and pitch angles of the aerial work platform, and the leakage current of the aerial work equipment, respectively.

7. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 1, characterized in that, VR display interaction devices have two operation modes: manual touch and voice control. The manual touch mode is used by operators to start and debug the system, while the voice control mode is used by operators to switch the vision system and adjust the viewing angle during operation.

8. The real-time interactive operation system for a master-slave teleoperated robot in a power distribution network as described in claim 1, characterized in that, The relevant characteristics of the work object include the position, shape, texture, and color of the work object.

9. An interaction method based on a real-time interactive operation system for a master-slave teleoperated robot in a power distribution network according to any one of claims 1-8, characterized in that, include: VR display and interaction devices generate corresponding control commands through voice control commands or operator movements to control the high-altitude vision module, tool vision module, and ground monitoring module to adjust or switch vision. VR display and interaction devices stitch together and merge video information received from high-altitude vision modules, tool vision modules, and ground monitoring modules to form a virtual scene based on digital twins. In the virtual scene based on digital twins, a mapping relationship between power distribution lines, robots, work tools, and actual working conditions is constructed to complete work tasks in the virtual scene and realize remote operation.

Images