Multi-person and multi-unmanned equipment sharing control system based on UDP (User Datagram Protocol) and use method

Abstract

The invention provides a multi-person and multi-unmanned equipment sharing control system based on a UDP (User Datagram Protocol) and a use method, and belongs to the technical field of communication. The control system comprises an unmanned device end and a tablet operation end, and comprises (a) an IP address dynamic discovery unit module used for realizing heartbeat information interaction regularly sent between the tablet operation end and the unmanned device end in a multicast mode; (b) a permission application and handover unit module which allows an operator to check a list of current online unmanned equipment and a name of a controller through a tablet terminal, and sends a control instruction to bind the unmanned equipment under the condition of permission; and (c) a control and sensor data return unit module for controlling the unmanned equipment through point-to-point operation. When one person operates a plurality of unmanned devices, a plurality of remote controllers do not need to be carried, and the operation of the plurality of unmanned devices can be realized at the same time only by carrying one tablet computer (which can be matched with a tablet computer with a rocker).

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a multi-personnel and multi-unmanned equipment sharing control system based on the UDP transmission protocol and a use method thereof. Background Art

[0002] The existing unmanned equipment control usually adopts a one-to-one connection between the remote controller and the unmanned equipment (such as drones, unmanned ships, unmanned vehicles, robot dogs, etc.). Therefore, there are as many remote controllers as there are unmanned equipment. When carrying multiple unmanned equipment outside, you must carry multiple corresponding remote controllers, which is easy to get confused. When one person operates multiple unmanned equipment, he needs to frequently switch between multiple remote controllers. This method is not convenient for managing and controlling multiple unmanned equipment.

[0003] The initial connection between the existing unmanned device remote controller and the unmanned device requires manual assistance: either the remote controller and the unmanned device are paired; or the IP address of the unmanned device needs to be obtained manually in advance, and then the IP address is recorded on the remote control end and communicated using a fixed IP address. However, when the device is added to a new local area network or the cache of the ad hoc network device is cleared, the IP address of the unmanned device will change, and the new IP address needs to be recorded on the remote control device end. In short, the existing device-to-device communication address acquisition is cumbersome.

[0004] Existing unmanned equipment requires remote control to be configured in advance, and equipment operators must carry the remote control when operating the unmanned equipment. However, it becomes a problem for people at a long distance (hundreds of meters to 10 kilometers) to operate the unmanned equipment. The latter needs to get the former's remote control to take turns operating the unmanned equipment.

[0005] Finally, although existing unmanned equipment can meet the needs of two remote controllers controlling one unmanned equipment at the same time, it is unable to flexibly and dynamically authorize different components or payloads of the unmanned equipment separately. Summary of the invention

[0006] The present invention effectively solves the above problems:

[0007] To address the problem that multiple unmanned equipment remote controls are easily confused and difficult to manage: In the present invention, when one person operates multiple unmanned equipment, there is no need to carry multiple remote controls. Only one tablet (which can be paired with a tablet with a joystick) is needed to operate multiple unmanned equipment at the same time.

[0008] The present invention mainly includes the following three unit modules:

[0009] 1IP address dynamic discovery unit module

[0010] The tablet operation terminal and the unmanned device terminal both send heartbeat information regularly in multicast mode, and the heartbeat information is accompanied by their respective IP addresses and identity information; the tablet operation terminal sends the heartbeat information with the number of the unmanned device being controlled; the tablet operation terminal receives the above heartbeat information and integrates the following data tables, including the online unmanned device list and its IP address mapping relationship table, the other online tablet operation terminals and their IP address mapping relationship table, and the unmanned device controller list. When the unmanned device cannot receive the heartbeat information sent by the control tablet terminal within several consecutive heartbeat cycles, the control relationship between it and the tablet is terminated, and its idle message is multicasted to the outside. Other tablets receive the multicast idle message of the unmanned device, modify the local unmanned device controller list, and set the unmanned device status to idle; when a new tablet comes online in the network, it immediately sends out multicast information to inform that it is online; other tablets receive the online multicast of the new tablet, and the tablet that is controlling the unmanned device sends its own identity and the identity of the controlled unmanned device to the new tablet, so that the new tablet can immediately obtain the current control relationship when it comes online; when the network status of the tablet or unmanned device changes (switching LAN or disconnecting and reconnecting), the UDP receiving module is automatically restarted to dynamically perceive the latest IP addresses of other devices.

[0011] 2. Control system permission application and handover unit module

[0012] The operator can check the list of currently online unmanned devices and the name of the controller by operating the tablet; the operator can check the status of the online unmanned devices on the tablet. If the unmanned device is idle, the operator can immediately send control instructions to the unmanned device through the tablet, and the unmanned device tablet and the unmanned device are successfully bound; the operator can check the status of the online unmanned device on the tablet. If the unmanned device is in a controlled state, the operator can send a point-to-point application to the tablet of the operator of the unmanned device through the tablet; when the current controller of the unmanned device receives a control application from others, he can click to agree or reject on the tablet operation end and send a point-to-point reply to the applicant; the applicant receives the reply, and if the controller of the unmanned device agrees, the control instruction is sent to the unmanned device, and a multicast message is sent at the same time that the unmanned device is controlled by himself; other personnel's tablets receive the unmanned device authority handover information and refresh the local unmanned device controller list; the operator of the unmanned device can actively release the control relationship of the unmanned device and send a multicast message. The person who is controlled by sending control instructions or applying for handover to others is the master controller. The master controller can actively authorize others to perform auxiliary control on different modules or operating units of the unmanned equipment (such as flight controller, gimbal, shooting unit, throwing module, robotic arm, etc.). The specific operation method is that the master controller selects one or more modules in the authorization management interface, and then selects a suitable operator from the currently online personnel list for authorization. The authorized auxiliary operator obtains the operating authority of the module authorized to operate, and so on. The master controller can flexibly authorize different modules of the equipment to multiple auxiliary operators.

[0013] 3. Control system control and sensor data feedback unit module

[0014] Operators can use the tablet to control the movement and lifting of unmanned equipment, adjust the posture of the unmanned equipment, adjust the speed, and control the direction of the gimbal through point-to-point operations; operators can edit routes on the tablet and send route tasks to the unmanned equipment. The unmanned equipment calculates the yaw angle based on the current position and the longitude and latitude of the waypoints in the route, and automatically adjusts the yaw angle to fly to the next waypoint, thereby realizing autonomous route flight of the unmanned equipment; the flight status data of the unmanned equipment, such as longitude and latitude, altitude, yaw angle, speed, flight time, GPS signal strength, battery power, gimbal angle, robotic arm status and other information are sent to the operating tablet in a point-to-point manner, and the tablet displays the above status information to the operator; the unmanned equipment flight video stream is sent to the tablet in a point-to-point manner, and the tablet displays the video screen to the operator.

[0015] The present invention has the following beneficial effects:

[0016] (1) In the present invention, the tablet operating terminal can dynamically sense the IP address of the unmanned device and can display the names of multiple unmanned devices in the form of a list on the tablet. The user can easily switch the unmanned devices for operation and can view the sensor information and returned videos of multiple unmanned devices on one screen at the same time.

[0017] (2) In the present invention, two people who are at a long distance can click the permission application button through the tablets they carry, and the other party can click to agree to the permission transfer in the tablet operation section to realize the function of taking turns to operate the same unmanned device.

[0018] (3) The present invention proposes a dynamically divisible authorization management method for different control units, which enables the master controller to select different operating units (such as flight controller, gimbal, shooting unit, throwing module, robotic arm, etc.) and actively authorize different people to operate on their respective tablet remote control terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of an operator of the present invention operating one or more unmanned devices through a tablet operation terminal and authority transfer;

[0020] Figure 2 It is a schematic block diagram of the functional modules of the unmanned device end and the tablet operation end of the present invention;

[0021] Figure 3 It is a schematic diagram of the interactive operation process between the unmanned device end and the tablet operation end of the present invention;

[0022] Figure 4 It is a schematic diagram of the interaction process between multiple tablet operating terminals of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0025] The hardware components of the system described in the present invention include: multiple unmanned equipment (such as drones, unmanned vehicles, robot dogs, unmanned ships), the unmanned equipment is equipped with a development board (such as Raspberry Pi, Jetson nano); multiple operating tablets pre-installed with Android system, the tablets are (such as Huawei tablets, RK3588 tablets or other tablets with joysticks); multiple radio stations.

[0026] The networking method of the present invention is to perform automatic mesh networking through a radio station, which can transmit WiFi signals. The communication method between the development board carried by the unmanned device (hereinafter collectively referred to as the unmanned device end) and the operation tablet (hereinafter collectively referred to as the tablet operation end) is UDP communication within the local area network; if there is a message for a specific target, it is sent in unicast mode; if there is no message for a specific target, it is sent in multicast mode.

[0027] Description of the interactive communication process between the tablet and the unmanned device:

[0028] 1 After the unmanned device is started and connected to the LAN, it automatically starts UDP listening to a port and sends startup information to other devices in the LAN in multicast mode. The content of the information sent includes the number and name of the unmanned device, so as to inform the tablet operation terminal in the LAN that it has come online.

[0029] 2. After the tablet operation terminal is started, it automatically starts UDP listening to a certain port. When receiving the startup message sent by the unmanned device terminal, it inserts the device number, name and IP address in the received message into the online unmanned device mapping table.

[0030] 3 The unmanned device regularly multicasts heartbeat information to the tablet operation terminal every 5 seconds as a keep-alive message.

[0031] 4. If the tablet operation terminal receives the heartbeat information from the unmanned device, it will refresh the last heartbeat time of the device in the online device table; if it does not receive the heartbeat information of an unmanned device for 30 consecutive seconds, it will determine that the unmanned device is offline and remove the unmanned device from the online device mapping table.

[0032] 5 The operator selects an idle unmanned device on the tablet operation terminal, clicks the connect button, and sends a connection instruction to the corresponding unmanned device in unicast mode. The information includes the name and code of the tablet operation terminal.

[0033] 6 The unmanned device receives the connection command and determines whether it is in an idle state. If it is in an idle state, it saves the device code that sent the connection command and replies to the IP address of the tablet operating terminal that sent the connection command that the connection is successful; otherwise, it replies with a connection failure message.

[0034] 7 The tablet receives the connection response information from the unmanned device. If the connection is successful, the status of the unmanned device in the online unmanned device mapping table is changed to "I control".

[0035] 8 After the connection is successful, the operator can send operation instructions to the unmanned device end in unicast mode through the tablet operation end, with the tablet code attached. Specific operation instructions include but are not limited to start, forward, backward, turn left, turn right, fly left, fly right, ascend, descend, stop, take off, land, return, accelerate, decelerate, send route, start route, cancel route, change gimbal pitch angle, roll angle, yaw angle, take pictures, set exposure, loudspeaker related instructions, robot arm related instructions, throwing, etc.;

[0036] 9 After receiving the operation instruction, the unmanned device first checks whether the code of the tablet operation terminal that sends the operation instruction is the code of the tablet operation terminal that is in a successfully connected state. If the verification is successful, the corresponding operation instruction is executed. If the verification fails, an illegal operation message is sent to the tablet operation terminal that sends the operation instruction in a unicast manner.

[0037] 10 The flight controller information, gimbal information, battery power information and various sensor module information generated by the unmanned equipment during operation are continuously sent to the corresponding tablet operation terminal in a unicast manner.

[0038] 11 The tablet operation terminal receives the above information and displays it to the operator in a graphical visualization manner.

[0039] 12 When two operators complete the permission application and handover through their respective tablets, the tablet operation terminal that applied for permission sends a change operator instruction in unicast mode, along with the tablet code, to the unmanned device terminal that performs the permission handover.

[0040] 13 The unmanned equipment receives the above-mentioned instruction to change the operator and refreshes the previously saved tablet code.

[0041] 14 The operator sends the video viewing instruction to the unmanned device terminal in unicast mode through the tablet operation terminal.

[0042] 15 After receiving the video viewing instruction, the unmanned device stores the code and IP address of the tablet operation terminal in the device list for viewing the video.

[0043] 16 The operator sends a command to cancel viewing the video to the unmanned device via the tablet operation terminal in unicast mode.

[0044] 17 After receiving the instruction to cancel video viewing, the unmanned device deletes the IP address of the tablet operation terminal from the list.

[0045] 18 The unmanned device sends video data of each frame of the video screen in unicast mode to the IP address in the list of videos to be viewed.

[0046] 19 The tablet operation terminal displays the video frame data in a BitMap format for the operator to view.

[0047] 20 The tablet operation terminal periodically multicasts heartbeat information to the unmanned device terminal every 5 seconds, along with the tablet code.

[0048] 21 After receiving the heartbeat information from the tablet operation terminal, the unmanned device terminal updates the last heartbeat time of the corresponding tablet. If the heartbeat information from the tablet operation terminal is not received for 30 consecutive seconds, it is determined that the tablet has been offline, and the connection with the tablet operation terminal is automatically disconnected, and the information that the unmanned device is in idle state is multicast to the other tablets.

[0049] 22 After using the unmanned device, the operator needs to actively release the connection with the unmanned device, select the unmanned device, click the release button, and send the release control command information to the corresponding unmanned device in unicast mode.

[0050] 23 After receiving the above release control instruction, the unmanned device sets the state of the device to idle.

[0051] Description of the interactive communication process between tablet terminals:

[0052] 1 The operator can interact with the tablet operation terminal (hereinafter referred to as the current tablet) and the operation tablet terminals used by other personnel (hereinafter referred to as other tablets). After the current tablet goes online (connects to the local area network), it sends multicast data to other tablets to inform them that they are online.

[0053] 2 After receiving the online information of the current tablet, other tablets check whether they are controlled by unmanned devices (one or more). If so, they respond to the current tablet with the information of the unmanned devices they control, including the tablet code and the code of the unmanned device.

[0054] 3 After the current tablet receives the online information sent by other tablets, it refreshes the control relationship between the tablet and the unmanned device to the local list

[0055] 4. Multicast heartbeat information to other tablets every 5 seconds, along with the tablet code.

[0056] 5. Other tablets receive the heartbeat information, update the online device list, and record the last heartbeat time of other tablets.

[0057] 6 If no heartbeat information is received from other tablets for 30 consecutive seconds, the other tablets are considered offline and the tablet is removed from the online device list.

[0058] 7. In a peer-to-peer manner, other tablets also multicast heartbeat information to the current tablet every 5 seconds. The logic of the current tablet receiving and processing heartbeat information is similar to the above content and will not be repeated here.

[0059] 8 The operator checks the list of online unmanned devices. When the unmanned device is idle, the operator can directly click the connection button to establish a connection with the unmanned device. After the connection is successful, the current tablet sends a multicast message to inform other tablets that it has established a connection with the corresponding unmanned device.

[0060] 9 After receiving the connection establishment information from the current tablet multicast, other tablets refresh the local online unmanned device list and change the status of the unmanned device to "other controlled"

[0061] 10 When the current tablet finds that the unmanned device has been controlled by someone else, the operator can click the Apply for Control button. The current tablet sends the application control command to the operating tablet of the requested unmanned device in unicast mode.

[0062] 11 After the operating tablet of the unmanned equipment receives the application control instruction, a pop-up window displays a prompt message to the current operator, and the operator either clicks to agree or clicks to reject.

[0063] 12 The current tablet receives the response information of other tablets to its application for controlling the unmanned device, checks if the other party agrees to the application, updates the local online unmanned device list, changes the state of the unmanned device to "I control", and sends the new controller identity of the unmanned device to other tablets in multicast mode

[0064] 13 Other tablets receive the information about changing the controller of the unmanned device sent by the current tablet, refresh the local online unmanned device list, and change the unmanned device to the new controller.

[0065] 14After the operator has finished using the unmanned device, he / she needs to actively release the connection with the unmanned device, select the unmanned device, click the release button, and notify other tablets via multicast.

[0066] 15 After receiving the information that the current tablet releases the unmanned device, the other tablets refresh the local online unmanned device list and reset the unmanned device status to idle.

Claims

1. A multi-person and multi-device sharing control system based on UDP transmission protocol, including an unmanned device terminal and a tablet operation terminal, characterized in that: include: (a) An IP address dynamic discovery unit module, which is used to implement the heartbeat information interaction between the tablet operation terminal and the unmanned device terminal by multicasting, and integrate the online unmanned device list and its IP address mapping relationship table, the other tablet operation terminal list and its IP address mapping relationship table, and the unmanned device controller list on the tablet operation terminal; (b) The permission application and handover unit module allows the operator to view the list of currently online unmanned devices and the name of the controller through the tablet, and send control commands to bind the unmanned devices if they have permission; (c) Control and sensor data feedback unit module, which controls the unmanned equipment through point-to-point operation.

2. The control system according to claim 1, characterized in that: The IP address dynamic discovery unit module further includes a function for releasing the control relationship between the unmanned device and the control tablet end and multicasting its idle message to the outside when the unmanned device cannot receive the heartbeat information sent by the control tablet end within several consecutive heartbeat cycles.

3. The control system according to claim 1, characterized in that: The permission application and handover unit module further includes a function for, when the current controller of the unmanned device receives a control application from another person, clicking to agree or reject on the tablet operating terminal and sending a reply to the applicant.

4. The control system according to claim 1, characterized in that: The control and sensor data feedback unit module further includes different modules for operators to control unmanned equipment such as pan-tilt heads and robotic arms through point-to-point operations on a tablet.

5. The control system according to claim 1, characterized in that: The control system further includes a tablet operation terminal code for verifying the tablet operation terminal code that sends the operation instruction after the unmanned equipment terminal receives the operation instruction, so as to confirm whether it is in a successful connection state.

6. The control system according to claim 4, characterized in that: The control system further includes a function for the master controller to select one or more modules on the authorization management interface, and then select a suitable operator from the current online personnel list for authorization.

7. The control system according to claim 6, characterized in that: The authorized auxiliary operator obtains the operating authority of the module that is authorized to operate, and the main controller can flexibly distribute the authorization of different modules of the equipment to multiple auxiliary operators.

8. The control system according to claim 1, characterized in that: The point-to-point operation for controlling the unmanned equipment includes controlling the movement and lifting of the unmanned equipment, adjusting the attitude and speed of the aircraft, controlling the direction of the gimbal, editing the route, autonomous route flight, and returning flight status data and video streams.

9. A method for using a control system, applied to the system according to claims 1 to 7, comprising the following steps: (a) The operator dynamically senses the IP address of the unmanned device through the tablet; (b) Send control commands through the tablet to bind the unmanned device; (c) Receive flight status data and video streams of unmanned equipment through the tablet; (d) Operators at a long distance are allowed to transfer permissions by clicking the permission button on the tablet, so that they can take turns operating the same unmanned equipment.

Images