Remote control method, system and equipment for unmanned ship and medium

Abstract

The invention discloses a remote control method, system and device for an unmanned ship and a medium, and relates to the technical field of remote control, and the method comprises the steps: converting a control instruction sent by a shore computer keyboard into a parameter variable which can be transmitted by a communication protocol MQTT through a monitoring program module, converting the obtained parameter variable into a control signal through a Raspberry Pi microcomputer carried by the unmanned ship, and controlling a motor carried by the unmanned ship to make a corresponding control action through the control signal; moving posture and position data of the unmanned ship in the actual environment are converted into parameter variables capable of being transmitted by MQTT through a message transmission module paho-mqtt, and the obtained parameter variables are mapped to a three-dimensional model of the unmanned ship and a three-dimensional model of the navigation environment of the unmanned ship through a shoreside computer. According to the invention, visual control can be realized under the condition of low-broadband communication.

Description

Technical Field

[0001] The present invention relates to the field of remote control technology, and in particular to a remote control method, system, equipment and medium for an unmanned ship. Background Art

[0002] As human beings continue to expand their use of the ocean, the demand for shipping and scientific research has also increased significantly, and unmanned ships have become popular due to their unique advantages. Compared with traditional manned ships, unmanned ships have better safety, stronger passability and environmental adaptability, lower maintenance costs and greater carrying capacity.

[0003] In the prior art, the remote control of the unmanned boat relies on the real-time transmission of video signals, and the operating status of the unmanned boat is monitored by high-definition video signals.

[0004] However, the video signal transmission technology used in remote control will occupy a large amount of bandwidth, causing network congestion and unstable transmission. Summary of the invention

[0005] The embodiments of the present invention provide a remote control method, system, device and medium for an unmanned ship, which can solve the problem in the prior art that remote control occupies a large amount of bandwidth.

[0006] An embodiment of the present invention provides a remote control method for an unmanned ship, comprising the following steps: establishing a three-dimensional model of the unmanned ship and the unmanned ship's navigation environment; and collecting motion posture and position data of the unmanned ship in an actual environment; converting control instructions into parameter variables that can be transmitted by a communication protocol MQTT by monitoring keyboard input, converting the acquired parameter variables into control signals, and controlling the unmanned ship to navigate in the actual environment and perform corresponding control actions through the control signals; converting the motion posture and position data during navigation into parameter variables that can be transmitted by MQTT, and mapping the acquired parameter variables to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment using digital twin technology, so that the three-dimensional model of the unmanned ship can display the motion posture and position of the unmanned ship when it navigates according to the control action in the three-dimensional model of the unmanned ship's navigation environment.

[0007] Furthermore, the control instructions sent from the shore computer keyboard are converted into parameter variables that can be transmitted by the communication protocol MQTT using a listening program module. The specific steps include: when it is detected that the W or S key of the shore computer is pressed, the host speed value used to control the speed of the unmanned boat motor is added or subtracted by a preset speed change; when it is detected that the A or D key of the shore computer is pressed, the dual-machine speed difference used to control the steering of the unmanned boat motor is added or subtracted by a predetermined speed change; and the changed host speed value and the dual-machine speed difference are written into the MQTT sending function.

[0008] Furthermore, the message delivery module runs in a loop every 0.1 seconds.

[0009] Furthermore, the parameter variables obtained are mapped to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment by the shore computer, and the specific steps include: the parameter variables obtained include the actual longitude and latitude data of the unmanned ship measured by the GPS carried by the unmanned ship, and the attitude data of the unmanned ship measured by the open source flight control Pixhawk; the actual longitude and latitude data between two points and the coordinate distance between the corresponding two points in the three-dimensional model of the unmanned ship's navigation environment are measured to obtain a proportional coefficient; the actual longitude and latitude data of the unmanned ship are mapped in the three-dimensional model of the unmanned ship's navigation environment according to the proportional coefficient, and the attitude data of the unmanned ship is mapped on the three-dimensional model of the unmanned ship.

[0010] Furthermore, the specific steps of establishing a three-dimensional model of the unmanned ship and its navigation environment include: using a drone to take multiple photos of the navigation environment; capturing environmental data in the environmental photos through the real-scene capture software ContextCapture to establish a three-dimensional model of the navigation environment; and using the ship design software Delftship and Rhino software to establish a three-dimensional model of the unmanned ship.

[0011] An embodiment of the present invention provides a remote control system for an unmanned ship, comprising:

[0012] A model building module is used to build a three-dimensional model of the unmanned ship and the unmanned ship's navigation environment; and to collect the motion posture and position data of the unmanned ship in the actual environment; a remote control module is used to convert the control instructions sent by the shore computer keyboard into parameter variables that can be transmitted by the communication protocol MQTT using a keyboard event listener, and convert the acquired parameter variables into control signals through the Raspberry Pi microcomputer carried by the unmanned ship, and control the unmanned ship to navigate in the actual environment and make corresponding control actions through the control signals; an unmanned ship visualization module is used to convert the motion posture and position data during navigation into parameter variables that can be transmitted by MQTT through the program library paho-mqtt, and map the acquired parameter variables to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment using digital twin technology through the shore computer, so that the three-dimensional model of the unmanned ship can display the motion posture and position of the unmanned ship when it navigates according to the control action in the three-dimensional model of the unmanned ship's navigation environment.

[0013] An embodiment of the present invention provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and the processor implements the above-mentioned remote control method of an unmanned ship when executing the computer program.

[0014] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the remote control method of an unmanned ship described above is implemented.

[0015] The embodiments of the present invention provide a remote control method, system, device and medium for an unmanned ship. Compared with the prior art, the advantages thereof are as follows:

[0016] The control instructions are converted into parameter variables that can be transmitted by MQTT by monitoring keyboard input and transmitted, the unmanned ship is controlled to move, and then the motion posture and position data of the unmanned ship in the actual environment are converted into parameter variables that can be transmitted by MQTT and transmitted, and mapped to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment using digital twin technology, and the three-dimensional model is updated. The communication protocol MQTT used in the transmission process is low-bandwidth communication. Finally, the visual control of the unmanned ship under low-bandwidth communication is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a control method provided by an embodiment of the present invention;

[0018] Figure 2 A technical roadmap provided for embodiments of the present invention;

[0019] Figure 3 A shore visualization system provided by an embodiment of the present invention;

[0020] Figure 4 A remote control flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0022] See also Figure 1 The embodiment of the present invention provides a remote control method for an unmanned ship, comprising the following steps:

[0023] Step 1: Create a three-dimensional model of the unmanned ship and its navigation environment.

[0024] The detailed operation steps are as follows: Use a drone equipped with an aerial camera to photograph the surveyed land. First, conduct an on-site survey of the survey area to determine the appropriate take-off point and shooting angle, and set the flight route of the drone according to the survey results. Secondly, the drone flies along the preset route, uses a high-definition camera to take environmental photos during the flight, and records GPS coordinates, altitude, and attitude information in real time. Thirdly, use the professional software ContextCapture to process the collected image data, including alignment and stitching. The software calculates the three-dimensional coordinates of each point by analyzing the overlapping parts between the images and the camera position information, thereby generating a three-dimensional point cloud, and then optimizes and refines the three-dimensional model. Finally, in order to reduce the rendering and loading pressure of the web platform, use MeshM i xer software to compress and fuse the triangular faces of the model mesh to reduce the number of meshes in the navigation environment model while maintaining the visual effect of the model. Finally, load the optimized model to the web platform for fast rendering and display. Using Deletesh ip, a professional software for ship design and stability analysis, Rhino software establishes a three-dimensional model of the ship, such as Figure 2-Figure 3 shown.

[0025] Step 2: Use the listening program module to convert the control instructions sent from the computer keyboard on the shore into parameter variables that can be transmitted by the communication protocol MQTT (MQ Telemetry Transport). The Raspberry Pi microcomputer on the unmanned boat converts the acquired parameter variables into control signals, and controls the motor on the unmanned boat to make corresponding control actions through the control signals.

[0026] The actual method of control is explained as follows:

[0027] A program is run on the computer on the shore, and there is a keyboard event listener in the program.

[0028] document.addEventL i stener('keydown', funct i on(event){

[0029] var key = event.key;}

[0030] If it is detected that the W or S button is pressed, the original host speed value initial Thrott le is added or subtracted by the predetermined speed change value ValueRange, and assigned to a new host speed variable newThrott le. Similarly, if it is detected that the A or D button is pressed, the original dual-machine speed difference initial Divers ion is added or subtracted by the predetermined speed change value ValueRange, and assigned to a new dual-machine speed difference newD ivers i on. Taking the "W" button as an example, the code is as follows.

[0031] if(key==='W'||key==='w'){

[0032] newThrott le=in it ial Thrott l e+Va l ueRange;

[0033] }

[0034] if(key==='W'||key==='w'||key==='S'||key==='s'){

[0035] newThrott le=in it ial Thrott le;

[0036] }

[0037] if(newThrott le!==thrott le||newD ivers i on!==d ivers ion){

[0038] thrott le = newThrott le;

[0039] di vers i on=newDi vers i on;

[0040] updateAndPr i ntData();

[0041] }

[0042] Through this series of programs, the instructions issued by the keyboard can be written into the sending function of the MQTT protocol and sent to the transit server in the cloud. After that, the transit server will forward the received information downward. At this time, the Raspberry Pi on the ship side computer with the preset receiving subject can receive the information, complete further information processing and control signal transmission inside it, and finally act on the motor of the unmanned boat to control the movement of the unmanned boat, such as Figure 4 shown.

[0043] Step 3: Collect the motion posture and position data of the unmanned boat in the actual environment. The motion posture and position data of the unmanned boat in the actual environment are converted into parameter variables that can be transmitted by MQTT through the message transmission module paho-mqtt, and the acquired parameter variables are mapped to the three-dimensional model of the unmanned boat and the three-dimensional model of the unmanned boat's navigation environment through the shore computer.

[0044] Specifically, the attitude and GPS data of the unmanned ship are collected mainly through the Pixhawk flight control and its supporting GPS locator. Pixhawk integrates sensors including three-axis accelerometer and three-axis gyroscope, MS5611 barometer, LSM303D accelerometer and magnetic compass. Pixhawk is fixed on the ship and powered on. After the program is initialized, it can generate and record matching motion information in real time with the movement. The information carried inside Pixhawk is obtained through the DroneKit application program interface (API) carried by the Raspberry Pi microcomputer. The program is written in Python, calling two third-party libraries, DroneKit and paho-mqtt. The former is used to obtain information in the flight control, and the latter is used to send data from the unmanned ship to the cloud transfer server. The program's loop running frequency is set to once every 0.1 seconds. The shore section computer obtains the information sent back from the cloud server, maps the position data to the virtual space position coordinates of the unmanned ship's three-dimensional model, and maps the attitude data to the motion attitude of the unmanned ship's three-dimensional model. Measure the actual longitude and latitude information of two points in the navigation environment and the coordinates of the corresponding positions of the web page environment model, calculate the ratio coefficient between the actual longitude and latitude and the coordinates in the web page, so as to convert the actual position of the unmanned ship into the relative position of the unmanned ship model on the web page platform and realize the mapping of position information.

[0045] The present invention can realize remote driving of unmanned ships under low-bandwidth conditions. Through digital twin technology, the system designed by the present invention improves real-time performance and accuracy, and reduces bandwidth requirements by using the communication protocol MQTT. The system is developed using WebGL technology, which can visualize the navigation environment and ship status on the web page, providing remote operators with a direct and clear basis for control. In addition, the system is developed using open source frameworks and general hardware, which not only reduces costs, but also improves the scalability and accessibility of the system. Through these technical means, the present invention aims to provide a cost-effective, bandwidth-free and easy-to-operate solution for remote control of unmanned ships.

[0046] The beneficial effects of the present invention include:

[0047] 1. Panoramic simulation under low-bandwidth communication conditions.

[0048] The system design takes low-bandwidth environments into consideration and optimizes data transmission and processing to achieve remote control and panoramic simulation of unmanned ships under bandwidth-constrained conditions.

[0049] 2. Application of digital twin technology.

[0050] By using digital twin technology, the real-time data of the unmanned ship and its environment are mapped into the virtual space, realizing real-time monitoring and control of the status of the unmanned ship, and improving the real-time performance and accuracy of the system.

[0051] 3. A web-based panoramic simulation platform based on WebGL.

[0052] A panoramic simulation platform was built on the web page using WebGL and Three.js framework, which enabled the loading and rendering of three-dimensional models of unmanned ships and navigation environments, and provided an intuitive and interactive control interface.

[0053] An embodiment of the present invention provides a system, including:

[0054] The model building module is used to build a three-dimensional model of the unmanned ship and the unmanned ship's navigation environment; and collect the motion posture and position data of the unmanned ship in the actual environment. The remote control module is used to convert the control instructions sent by the computer keyboard on the shore into parameter variables that can be transmitted by the communication protocol MQTT using the keyboard event listener, and convert the acquired parameter variables into control signals through the Raspberry Pi microcomputer carried by the unmanned ship, and control the unmanned ship to navigate in the actual environment and make corresponding control actions through the control signals. The unmanned ship visualization module is used to convert the motion posture and position data during navigation into parameter variables that can be transmitted by MQTT through the program library paho-mqtt, and map the acquired parameter variables to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment using the digital twin technology through the shore computer, so that the three-dimensional model of the unmanned ship can display the motion posture and position of the unmanned ship when it is sailing according to the control action in the three-dimensional model of the unmanned ship's navigation environment.

[0055] An embodiment of the present invention provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and the processor implements the steps of a remote control method for an unmanned ship when executing the computer program.

[0056] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of a remote control method for an unmanned ship are implemented.

[0057] A specific embodiment is as follows:

[0058] 1. Establishing the navigation environment and 3D model of the ship: Use drones to take photos of the navigation environment. ContextCapture software reads the GPS coordinates, flight altitude, lens and fuselage tilt angles of the drone in the drone aerial images. By comparing and calculating thousands of photos in the same area, the photographed terrain is established into a 3D model. Using Delftship, Rhino software establishes a 3D model of the ship.

[0059] 2. Use JavaScript language and WebGL technology to develop a web platform, load the three-dimensional model of the unmanned ship and the navigation environment, and build a virtual navigation scene.

[0060] 3. Use Raspberry Pi, Pixhawk flight control module, sensors and other hardware to build an unmanned boat.

[0061] 4. Collect data using Python language.

[0062] 5. Use the MQTT communication protocol to transmit the status information of the unmanned ship to the shore in real time, and map it in the virtual environment of the web page in real time.

[0063] 6. According to the unmanned ship information in the virtual environment of the web page, the next step of control is carried out. The control signals of the keyboard and mouse are converted into data packets through the JavaScript program, and sent to the Raspberry Pi microcomputer of the real ship through the MQTT protocol to realize the sending of control instructions.

[0064] 7. Use digital twin technology to achieve mutual mapping between virtual and real data.

[0065] It has been verified that 10 status data packets are transmitted per second from the ship to the shore, and 3 status data packets are transmitted per second from the shore to the ship, and the bandwidth occupancy does not exceed 2.5KB / s. It can realize remote driving of unmanned ships with low bandwidth.

[0066] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A remote control method for an unmanned ship, characterized in that: The following steps are involved: Establish a three-dimensional model of the unmanned ship and its navigation environment; and collect the motion posture and position data of the unmanned ship in the actual environment; The control instructions are converted into parameter variables that can be transmitted by the communication protocol MQTT by monitoring keyboard input, and the obtained parameter variables are converted into control signals. The control signals are used to control the unmanned ship to navigate in the actual environment and make corresponding control actions; The motion posture and position data during navigation are converted into parameter variables that can be transmitted by MQTT, and the acquired parameter variables are mapped to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment using digital twin technology, so that the three-dimensional model of the unmanned ship can display the motion posture and position of the unmanned ship when sailing according to the control action in the three-dimensional model of the unmanned ship's navigation environment.

2. A remote control method for an unmanned ship as claimed in claim 1, characterized in that: The control instructions sent by the computer keyboard on the shore are converted into parameter variables that can be transmitted by the communication protocol MQTT using the monitoring program module, and the specific steps include: When the W or S key of the shore computer is detected to be pressed, the host speed value used to control the speed of the unmanned boat motor is added to or subtracted from the preset speed change; When the A or D key of the shore computer is detected to be pressed, the speed difference between the two engines used to control the steering of the motor of the unmanned boat is added or subtracted by a predetermined speed change; The changed host speed value and the difference between the two machine speeds are written into the MQTT sending function.

3. The remote control method of an unmanned ship according to claim 1, characterized in that: The message delivery module runs in a loop every 0.1 seconds.

4. The remote control method of an unmanned ship according to claim 1, characterized in that: The method of mapping the acquired parameter variables to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment by the shore computer specifically includes: The acquired parameter variables include the actual latitude and longitude data of the unmanned ship measured by the GPS carried by the unmanned ship, and the attitude data of the unmanned ship measured by the open source flight control Pixhawk; Measure the actual longitude and latitude data between two points and the coordinate distance between the two corresponding points in the three-dimensional model of the unmanned ship's navigation environment to obtain a proportional coefficient; The actual latitude and longitude data of the unmanned ship are mapped onto the three-dimensional model of the unmanned ship's navigation environment according to the proportional coefficient, and the unmanned ship's attitude data are mapped onto the three-dimensional model of the unmanned ship.

5. The remote control method of an unmanned ship according to claim 1, characterized in that: The specific steps of establishing the three-dimensional model of the unmanned ship and the unmanned ship's navigation environment include: Use drones to take multiple photos of the navigation environment; The environmental data in the environmental photos are captured by the real-scene capture software ContextCapture to establish a 3D model of the navigation environment; The 3D model of the unmanned ship was built using the ship design software Delftship and Rhino software.

6. A remote control system for an unmanned ship, characterized in that: include: Model building module, used to build a three-dimensional model of the unmanned ship and the unmanned ship's navigation environment; And collect the motion posture and position data of the unmanned ship in the actual environment; The remote control module is used to convert the control instructions sent by the computer keyboard on the shore into parameter variables that can be transmitted by the communication protocol MQTT using the keyboard event listener, and convert the acquired parameter variables into control signals through the Raspberry Pi microcomputer carried by the unmanned boat, and control the unmanned boat to navigate in the actual environment and make corresponding control actions through the control signals; The unmanned ship visualization module is used to convert the motion posture and position data during navigation into parameter variables that can be transmitted by MQTT through the program library paho-mqtt, and map the acquired parameter variables to the three-dimensional model of the unmanned ship and the three-dimensional model of the unmanned ship's navigation environment through the shore computer using digital twin technology, so that the three-dimensional model of the unmanned ship can display the motion posture and position of the unmanned ship when sailing according to the control action in the three-dimensional model of the unmanned ship's navigation environment.

7. A computer device comprising: Memory and processor; The memory stores a computer program, wherein the processor implements a remote control method for an unmanned ship according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a remote control method for an unmanned ship according to any one of claims 1 to 5 is implemented.

Images