A system for real-time three-dimensional monitoring of the motion of a rail-mounted robot

Abstract

The present invention relates to a system for real-time three-dimensional monitoring of the motion of a rail-hanging robot. A user can directly load the client module of the system in the browser by operating a general WEB browser (such as Chrome or Firefox) on a PC or mobile phone to connect to the relevant pages of the robot through the network, without installing additional simulation software; the client module includes a three-dimensional motion monitoring client module and a three-dimensional environment editing client module, and the robot runs a three-dimensional motion monitoring server module and a three-dimensional environment editing server module; the three-dimensional motion monitoring server module is used to transmit the current motion state data of the robot to the three-dimensional motion monitoring client module, and control the actual motion of the robot; the three-dimensional environment editing server module is used to create a three-dimensional simulation environment configuration file with the three-dimensional environment editing client module. The beneficial effect is to observe the motion state of the rail-hanging robot in the environment in real time, control remotely, simply and intuitively, and reach it with one key.

Description

[Technical field]

[0001] The invention relates to the technical field of robots and image processing, and in particular to a system for three-dimensional real-time monitoring of the movement of a rail-hanging robot. [Background technology]

[0002] The rail-hanging (inspection) robot is a comprehensive system that integrates the robot body, wireless communication, and host computer management and monitoring system. The robot body has functions such as infrared thermal imaging, image shooting, environmental detection, wireless data transmission, and alarm indication. Relying on advanced visual recognition technology and intelligent sensor technology, it can collect and store data from various instruments and predict environmental hazards. The rail-hanging robot inspection can achieve 24-hour uninterrupted work and can replace manual patrol inspections, environmental monitoring, fault warnings, and image intelligent analysis in complex and dangerous environments. It is widely used in pipe corridors, tunnels, subways, chemical plants, power stations, airports, and other occasions.

[0003] Currently in the field of track-mounted robots, robots are usually managed through web pages in browsers, but there is no method to intuitively monitor the current three-dimensional motion state of the robot on the management interface. The robot can only observe the surrounding environment through the robot's own camera video screen, or can only indicate the current position of the robot through a two-dimensional plane diagram. For robot managers, it is impossible to intuitively monitor the motion state of the robot itself and its relationship with the environment. The robot manager can only infer the state through the local picture of the camera and cannot obtain global information.

[0004] In addition, if the robot needs to be manually controlled to move to a certain position on the track, the traditional method generally requires the manager to continuously operate the robot until it reaches the appropriate position. The position can be determined either through the image transmitted back by the camera on the robot, or the operator needs to stay at the robot site for on-site observation. The former often leads to inaccurate position judgment due to the limitation of the viewing angle, and the latter cannot truly achieve remote control.

[0005] The fourth issue of the Journal of Hebei University of Technology in 2015 published "A Method for Real-time Three-dimensional Motion Simulation of Robots". The authors Huang Xiaochen, Zhang Minglu, Zhang Xiaojun, Bai Feng, and Gao Han proposed a method for real-time three-dimensional motion simulation of robots to address the problems that the commonly used methods for real-time three-dimensional motion simulation of robots cannot be integrated with the control system development platform and lack algorithm support. This method uses Solid Works software as the carrier for displaying the robot's three-dimensional model, and uses Visual Studio C++ software for secondary development of the three-dimensional software, integrating the control system with the three-dimensional real-time simulation platform. It can not only issue control instructions to the robot, but also use the three-dimensional real-time simulation platform to simulate the robot's movement in real time. Research shows that the use of this real-time motion simulation method can use the feedback position data to drive the corresponding proportional three-dimensional model in real time while ensuring accuracy, and record the robot's motion data in real time, making the robot's movement more intuitive and accurate.

[0006] SLAM (Simultaneous Localization and Mapping) is a mature technology in the field of robotics. It can simultaneously estimate the state of the robot and build a model of its environment. The SLAM problem can be described as: the robot starts to move from an unknown position in an unknown environment, and locates itself according to the position estimation and map during the movement. At the same time, it builds an incremental map based on its own positioning to achieve autonomous positioning and navigation of the robot. "A Visual SLAM System for Dynamic Target Perception" discloses a robust dynamic target perception SLAM system (VDO-SLAM). The system uses semantic information to estimate the motion of rigid objects in the scene without prior information on the target shape or motion model. This method integrates the dynamic and static structures in the environment into a unified estimation framework, thereby achieving accurate estimation of the robot camera and spatiotemporal map. Application number: CN201910768775.6, invention name "A visual perception mapping algorithm and mobile robot based on SLAM", the disclosed algorithm includes obtaining image data of the environment of the area to be mapped and the speed data of the driving wheel; extracting features from the color image in the image data; after the speed data is calculated and processed according to the kinematic equation, the speed data is fused and optimized together with the depth image in the image data by minimizing the encoder and reprojection errors; the current frame pose of the robot is obtained; the current frame image data obtained when the robot walks in the area to be mapped is screened to obtain a key frame, and the color image of the key frame is used to identify and classify objects in the environment, and the classified dynamic objects are deleted from the key frame, and the static objects are sparsely processed, and local loop correction is performed; the pixel points of the color image and the corresponding depth image in the key frame are used as map points, and the map points are stored and mapped through an octree.

[0007] The present invention improves the system for three-dimensional real-time monitoring of the motion of the rail-hanging robot in a manner different from the above-mentioned prior art. [Summary of the invention]

[0008] The purpose of the present invention is to provide a system for real-time observation of the motion state of a rail-mounted robot in an environment, and for controlling the motion of the rail-mounted robot in three dimensions in a remote, simple, intuitive, and one-key manner.

[0009] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is a system for three-dimensional real-time monitoring of the movement of a rail-mounted robot, including a client module and a robot executed in a general WEB browser. The above-mentioned WEB browser end is just a general WEB browser running on various devices, which is used to load and execute the client module from the robot end; the above-mentioned client module includes a three-dimensional motion monitoring client module and a three-dimensional environment editing client module, and the above-mentioned robot runs a three-dimensional motion monitoring server module and a three-dimensional environment editing server module; the above-mentioned three-dimensional motion monitoring server module is responsible for docking with the three-dimensional motion monitoring client module, and is used to pass the robot's current motion state data to the three-dimensional motion monitoring client module, and to send the three-dimensional motion monitoring client module to the three-dimensional motion monitoring client module. The motion request generated is sent to the motion control unit inside the robot to control the actual motion of the robot; the above-mentioned 3D motion monitoring client module is used to obtain motion data from the 3D motion monitoring server module, display the state and position of the robot in the 3D simulation environment in real time, and transmit the specified position back to the 3D motion monitoring server module, thereby controlling the robot to move to the specified position; the above-mentioned 3D environment editing server module is responsible for connecting to the 3D environment editing client module, and is used to convert the simulation environment information created and modified by the 3D environment editing client module into a 3D simulation environment configuration file; the above-mentioned 3D environment editing client module is used to create and modify the simulation environment information, and transmit the data back to the 3D environment editing server module to save it as a 3D simulation environment configuration file.

[0010] Preferably, the system for three-dimensional real-time monitoring of the motion of the rail-hanging robot comprises the following interactive steps when used for three-dimensional real-time monitoring of the motion of the rail-hanging robot:

[0011] S1. The three-dimensional environment editing client module on the WEB browser side connects to the three-dimensional environment editing server module on the robot side to edit the three-dimensional simulation environment configuration file according to the actual environment, and saves it for use by the three-dimensional motion monitoring client module on the WEB browser side and the three-dimensional motion monitoring server module on the robot side;

[0012] S2, the three-dimensional motion monitoring client module on the WEB browser side connects to the three-dimensional motion monitoring server module on the robot side to obtain the three-dimensional simulation environment configuration file and the three-dimensional model files of each joint component of the actual robot, and draw the simulation environment and the simulation robot;

[0013] S3, the three-dimensional motion monitoring client module on the WEB browser side continuously obtains the position information of the actual robot on the track and the status of each joint component of the actual robot through the three-dimensional motion monitoring server module on the robot side. The three-dimensional motion monitoring client module on the WEB browser side continuously refreshes and draws the position and joint posture of the simulated robot in the simulation environment according to the position information of the actual robot on the track and the status of each joint component of the actual robot, so as to achieve complete synchronization between the simulation environment and the actual environment, and between the simulated robot and the actual robot;

[0014] S4, specifying position information in the simulation environment through the three-dimensional motion monitoring client module on the WEB browser side to control the motion of the simulation robot, and sending the position information as a control command to the three-dimensional motion monitoring server module on the robot side, and the three-dimensional motion monitoring server module on the robot side applies the control command to each joint component of the robot to realize the synchronous motion of the actual robot in the actual environment;

[0015] S5. Trigger step S3 or step S4 according to the actual needs of three-dimensional real-time monitoring of the movement of the rail-hanging robot.

[0016] Preferably, in the above-mentioned system for three-dimensional real-time monitoring of the movement of a rail-mounted robot, in step S1, a three-dimensional environment editing client module on a WEB browser side creates rail and cabinet environment elements according to the actual environment, and the position dimensions of the rail and the cabinet use actually measured values ​​to edit and obtain a three-dimensional simulation environment configuration file.

[0017] Preferably, in the above-mentioned system for three-dimensional real-time monitoring of the motion of a track-mounted robot, in step S4, the user uses a mouse to click on a specified position of the track in a simulation environment of a three-dimensional motion monitoring client module on a WEB browser side, and uses three-dimensional coordinate transformation to convert the specified position into position information of the actual track.

[0018] Preferably, in the above-mentioned system for three-dimensional real-time monitoring of the motion of a rail-mounted robot, the client module runs in a WEB browser without the need for additional software.

[0019] The present invention has the following beneficial effects: through the synchronous control and data interaction between the client software and the robot-side service module, the convenience and intuitiveness of the motion management and control of the rail-mounted robot are greatly increased, and one-click access reduces the training and learning cost of controlling and managing the rail-mounted robot; the simulation of three-dimensional motion does not require the installation of additional software such as SolidWorks, nor does it require a specific PC for installing client software. All software is deployed inside the robot, and the user only needs to use it on any device equipped with a general web browser. The client module is automatically loaded and run from the robot side by the WEB browser; the mapping of the environment is also different from the SLAM method. It does not require the use of sensors such as laser radar or cameras, but uses a visual manual method to generate a virtual three-dimensional environment. It is convenient and effective, does not require additional costs, and is particularly suitable for the rail-mounted robot environment.

Brief Description of the Drawings

[0020] Figure 1 It is a three-dimensional real-time monitoring system for the movement of rail-mounted robots.

[0021] Figure 2 The present invention is a schematic diagram of information interaction of functional modules of a software system for three-dimensional real-time monitoring of the motion of a rail-mounted robot in a WEB browser. [Specific implementation method]

[0022] The present invention will be further described below in conjunction with embodiments and with reference to the accompanying drawings.

[0023] In the present invention, the server runs inside the robot, which can be a conventional computer or embedded device and can provide network connection services. The client device can be any device that can connect to the network and has a general web browser installed, such as a general PC, mobile phone or pad, etc., without the need to install additional 3D simulation software.

[0024] Example 1

[0025] This embodiment implements a system for three-dimensional real-time monitoring of robot motion.

[0026] Attached Figure 1 This is a three-dimensional real-time monitoring system for the movement of the rail-mounted robot. Figure 1As shown, the present embodiment is a system for three-dimensional real-time monitoring of the motion of a rail-mounted robot, a client module and a robot executed in a universal WEB browser, the WEB browser end is just a universal WEB browser running on various devices, and is used to load and execute the client module from the robot end; the client module includes a three-dimensional motion monitoring client module and a three-dimensional environment editing client module, and the robot runs a three-dimensional motion monitoring server module and a three-dimensional environment editing server module inside; the three-dimensional motion monitoring server module is responsible for docking with the three-dimensional motion monitoring client module, and is used to transmit the current motion state data of the robot to the three-dimensional motion monitoring client module, and to send the motion request issued by the three-dimensional motion monitoring client module to the three-dimensional motion monitoring client module. The 3D motion monitoring client module is used to obtain motion data from the 3D motion monitoring server module, display the status and position of the robot in the 3D simulation environment in real time, and transmit the designated position back to the 3D motion monitoring server module, thereby controlling the robot to move to the designated position; the 3D environment editing server module is responsible for connecting to the 3D environment editing client module, and is used to convert the simulation environment information created and modified by the 3D environment editing client module into a 3D simulation environment configuration file; the 3D environment editing client module is used to create and modify the simulation environment information, and transmit the data back to the 3D environment editing server module to be saved as a 3D simulation environment configuration file.

[0027] Preferably, the system for three-dimensional real-time monitoring of the motion of the rail-hanging robot comprises the following interactive steps when used for three-dimensional real-time monitoring of the motion of the rail-hanging robot:

[0028] S1. The 3D environment editing client module on the WEB browser side connects to the 3D environment editing server module on the robot side to edit the 3D simulation environment configuration file according to the actual environment, and saves it for use by the 3D motion monitoring client module on the client software and the 3D motion monitoring server module on the robot side;

[0029] S2, the three-dimensional motion monitoring client module on the WEB browser side connects to the three-dimensional motion monitoring server module on the robot side to obtain the three-dimensional simulation environment configuration file and the three-dimensional model files of each joint component of the actual robot, and draw the simulation environment and the simulation robot;

[0030] S3, the three-dimensional motion monitoring client module on the WEB browser side continuously obtains the position information of the actual robot on the track and the status of each joint component of the actual robot through the three-dimensional motion monitoring server module on the robot side. The three-dimensional motion monitoring client module on the WEB browser side continuously refreshes and draws the position and joint posture of the simulated robot in the simulation environment according to the position information of the actual robot on the track and the status of each joint component of the actual robot, so as to achieve complete synchronization between the simulation environment and the actual environment, and between the simulated robot and the actual robot;

[0031] S4, specifying position information in the simulation environment through the three-dimensional motion monitoring client module on the WEB browser side to control the motion of the simulation robot, and sending the position information as a control command to the three-dimensional motion monitoring server module on the robot side, and the three-dimensional motion monitoring server module on the robot side applies the control command to each joint component of the robot to realize the synchronous motion of the actual robot in the actual environment;

[0032] S5. Trigger step S3 or step S4 according to the actual needs of three-dimensional real-time monitoring of the movement of the rail-hanging robot.

[0033] Preferably, in the above-mentioned system for three-dimensional real-time monitoring of the movement of a rail-mounted robot, in step S1, a three-dimensional environment editing client module on a WEB browser side creates rail and cabinet environment elements according to the actual environment, and the position dimensions of the rail and the cabinet use actually measured values ​​to edit and obtain a three-dimensional simulation environment configuration file.

[0034] Preferably, in the above-mentioned system for three-dimensional real-time monitoring of the motion of a track-mounted robot, in step S4, the user uses a mouse to click on a specified position of the track in a simulation environment of a three-dimensional motion monitoring client module on a WEB browser side, and uses three-dimensional coordinate transformation to convert the specified position into position information of the actual track.

[0035] Preferably, in the above-mentioned system for three-dimensional real-time monitoring of the movement of the rail-hanging robot, the client module runs in a WEB browser. The user operates a general WEB browser (such as Chrome or Firefox) on a PC or mobile phone to connect to the relevant pages of the robot through the network and can directly load and use the client module of the system in the browser without installing additional simulation software.

[0036] Example 2

[0037] This embodiment implements a software system for three-dimensional real-time monitoring of the motion of a rail-mounted robot in a WEB browser.

[0038] In view of the defects in the prior art, the technical problems to be solved by the software system for real-time three-dimensional monitoring of the movement of the rail-mounted robot in the browser in this embodiment are embodied in the following points: 1. The movement state of the rail-mounted robot body in the environment can be observed in real time from a bird's eye view in a simulated three-dimensional environment, including the position of the robot on the track, its relative position with the environment (mainly the cabinet), and the state of any movable joints of the robot, which are all consistent with the physical robot; 2. The manager can directly click on the position on the track in the three-dimensional environment to manually control the robot to run to that position. The operation is simple and intuitive, and it can be reached with one click, truly achieving remote control at a different location; the simulated three-dimensional environment can be rotated and scaled arbitrarily, and there are no blind spots for observation.

[0039] The viewing operation of the 3D environment is carried out on the web page of the browser without the need to install additional software; the editing of the 3D environment is also carried out on the browser, and environmental information such as tracks and cabinets can be created and edited by combining mouse drag-and-drop rotation with precise data settings, providing an accurate simulation of the actual environment.

[0040] The software system for three-dimensional real-time monitoring of the motion of a rail-hanging robot in a browser in this embodiment includes the following software modules:

[0041] Three-dimensional motion monitoring server module: runs inside the robot and is responsible for connecting to the three-dimensional motion monitoring client module; it can pass the robot's current motion status data to the three-dimensional motion monitoring client module; at the same time, it can send the motion request issued by the three-dimensional motion monitoring client module to the motion control unit inside the robot to control the actual movement of the robot.

[0042] 3D motion monitoring client module: runs in the web browser, obtains motion data from the 3D motion monitoring server module through the network, and displays the status and position of the robot in the simulation environment in real time; and can transmit the position specified by the mouse back to the 3D motion monitoring server module, thereby controlling the robot to move to the specified position.

[0043] 3D environment editing server module: runs inside the robot and is responsible for converting the environment information created and modified by the 3D environment editing client module into configuration files or parameters in memory. The 3D environment editing client module can also load existing environment configuration files through this module.

[0044] 3D environment editing client module: runs in the web browser, and can create and modify simulation environments such as tracks and cabinets by dragging and rotating the mouse, and transmit the data back to the 3D environment editing server module to save it as an environment configuration file.

[0045] The 3D motion monitoring server module and the 3D environment editing server module reside inside the robot. When there is no client connected, they do not operate. When a browser client is connected, they will establish a connection with the client and complete the interaction process. If multiple clients are connected at the same time, they will establish independent connections to complete the interaction. Figure 2 This is a schematic diagram of the information interaction of the functional modules of a software system that monitors the motion of a rail-mounted robot in real time in a three-dimensional manner in a WEB browser. Figure 2 As shown, the specific information interaction process of a software system for three-dimensional real-time monitoring of the motion of a rail-hanging robot in a browser in this embodiment is described as follows:

[0046] When the three-dimensional motion monitoring client module is connected to the server through the network, 1. First, the current environment configuration file and the three-dimensional model files of the robot's movable parts are obtained from the server, which are used to draw the three-dimensional simulation environment and the robot; 2. Then the three-dimensional motion monitoring server module continuously obtains the robot's position information on the track and the status of the robot's movable joints and sends them to the client module. The client module can draw the robot's position in the environment and its own joint posture based on this constantly refreshed information, thus achieving complete synchronization between the simulation environment and the actual environment; 3. Conversely, when the operator clicks the specified position of the track with the mouse in the simulation environment, the client module uses the three-dimensional coordinate transformation to convert the mouse click coordinates into the actual track position information, and sends it back to the server module as a control command, which is then forwarded by the server module to the actual control unit to realize the movement of the robot.

[0047] When the 3D environment editing client module is connected to the server through the network, 1. First, the environment configuration file that needs to be edited is obtained from the server. If it does not exist, it is empty. A 3D simulation environment to be edited (or an empty environment) can be generated based on the loaded configuration file; 2. Then the operator can use the mouse or enter specific values ​​on the client to create environmental elements such as tracks and cabinets. The position and size of the tracks and cabinets use the actual measured values ​​to ensure consistency. Each step of creation and modification will be transmitted back to the server module and saved in the memory; 3. After the command to save the file is issued, the server actually saves the environmental parameters as a configuration file, and this configuration file can be used by the 3D motion monitoring server / client module.

[0048] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in any computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0049] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A system for three-dimensional real-time monitoring of the motion of a rail-hanging robot, characterized in that: The system for real-time three-dimensional monitoring of the motion of the rail-hanging robot includes a client module executed in a general WEB browser and a robot. The WEB browser is just a general WEB browser running on various devices, which is used to load and execute the client module from the robot. The client module includes a three-dimensional motion monitoring client module and a three-dimensional environment editing client module. The robot runs a three-dimensional motion monitoring server module and a three-dimensional environment editing server module. The three-dimensional motion monitoring server module is responsible for docking with the three-dimensional motion monitoring client module, for transmitting the current motion state data of the robot to the three-dimensional motion monitoring client module, and for sending the motion request issued by the three-dimensional motion monitoring client module to the robot. The motion control unit inside the robot is used to control the actual motion of the robot; the three-dimensional motion monitoring client module is used to obtain motion data from the three-dimensional motion monitoring server module, display the state and position of the robot in the three-dimensional simulation environment in real time, and transmit the specified position back to the three-dimensional motion monitoring server module, thereby controlling the robot to move to the specified position; the three-dimensional environment editing server module is responsible for connecting to the three-dimensional environment editing client module, and is used to convert the simulation environment information created and modified by the three-dimensional environment editing client module into a three-dimensional simulation environment configuration file; the three-dimensional environment editing client module is used to create and modify the simulation environment information, and transmit the data back to the three-dimensional environment editing server module to save it as a three-dimensional simulation environment configuration file.

2. A system for three-dimensional real-time monitoring of the motion of a rail-mounted robot according to claim 1, characterized in that The system for three-dimensional real-time monitoring of the motion of the rail-hanging robot includes the following interactive steps when used for three-dimensional real-time monitoring of the motion of the rail-hanging robot: S1. The 3D environment editing client module on the WEB browser side connects to the 3D environment editing server module on the robot side to edit the 3D simulation environment configuration file according to the actual environment, and saves it for use by the 3D motion monitoring client module on the client software and the 3D motion monitoring server module on the robot side; S2, the three-dimensional motion monitoring client module on the WEB browser side connects to the three-dimensional motion monitoring server module on the robot side to obtain the three-dimensional simulation environment configuration file and the three-dimensional model files of each joint component of the actual robot, and draw the simulation environment and the simulation robot; S3, the three-dimensional motion monitoring client module on the WEB browser side continuously obtains the position information of the actual robot on the track and the status of each joint component of the actual robot through the three-dimensional motion monitoring server module on the robot side. The three-dimensional motion monitoring client module on the WEB browser side continuously refreshes and draws the position and joint posture of the simulated robot in the simulation environment according to the position information of the actual robot on the track and the status of each joint component of the actual robot, so as to achieve complete synchronization between the simulation environment and the actual environment, and between the simulated robot and the actual robot; S4, specifying position information in the simulation environment through the three-dimensional motion monitoring client module on the WEB browser side to control the motion of the simulation robot, and sending the position information as a control command to the three-dimensional motion monitoring server module on the robot side, and the three-dimensional motion monitoring server module on the robot side applies the control command to each joint component of the robot to realize the synchronous motion of the actual robot in the actual environment; S5. Trigger step S3 or step S4 according to the actual needs of three-dimensional real-time monitoring of the movement of the rail-hanging robot.

3. A system for three-dimensional real-time monitoring of the motion of a rail-mounted robot according to claim 2, characterized in that: In step S1, the three-dimensional environment editing client module on the WEB browser side creates track and cabinet environment elements according to the actual environment. The position and size of the track and the cabinet use the actual measured values ​​to edit and obtain the three-dimensional simulation environment configuration file.

4. A system for three-dimensional real-time monitoring of the motion of a rail-mounted robot according to claim 2, characterized in that: In step S4, the user uses a mouse to click on a designated position of the track in the simulation environment of the three-dimensional motion monitoring client module on the WEB browser side, and uses three-dimensional coordinate transformation to convert the designated position into position information of the actual track.

5. The system for three-dimensional real-time monitoring of the motion of a rail-mounted robot according to claim 1, characterized in that: The client module runs in a general WEB browser without the need for additional software.

Images