A dual-screen remote control data processing method and system for unmanned aerial vehicles

Abstract

The present invention relates to the field of drone control technology, and discloses a dual-screen remote control data processing method and system for drones, comprising: initiating a data acquisition request from a starting remote control to a starting drone via an input module and an IP proxy communication module; receiving response data information from the starting drone regarding the data acquisition request in the IP proxy communication module; forwarding the response data information to a data processor via the IP proxy communication module; forwarding the response data information to a USB data service communication module via the IP proxy communication module and a shared memory processing module; and then sending the response data information to a starting mobile device via a main control module and the USB data service communication module; returning data control information to the IP proxy communication module via the USB data service communication module and the shared memory processing module; and initiating a data control request to the starting drone via the IP proxy communication module. The present invention can achieve UI clicks and simultaneous multi-screen display.

Description

Technical Field

[0001] The present invention relates to a dual-screen remote controller data processing method and system applied to an unmanned aerial vehicle (UAV), belonging to the technical field of UAV control. Background Art

[0002] Nowadays, the remote controls of aerial photography drones are gradually tending to have built-in display screens. The image data taken by the aerial photography drone can be viewed on the display screen, and based on the image data, the image angle that the aerial photography drone should shoot next can be decided. The remote control can then send commands for viewing angle control such as rotation, translation, and lifting to the aerial photography drone, thereby controlling the aerial photography drone to take images and videos.

[0003] At present, in order to reduce the cost of remote control settings, most of the mid- and low-end remote control products on the market use displays without touch function, that is, they can only be used to display the image, status and other information of the drone, and there is no UI click interaction function. In addition, when the remote control comes with multiple displays, these displays cannot work at the same time, that is, only one display is working at the same time, that is, when different displays display different content, the user can only view the content on one of the displays during the same period of time.

[0004] Therefore, a solution is urgently needed to achieve UI tapping and multi-screen simultaneous display. Summary of the Invention

[0005] The present invention provides a dual-screen remote control data processing method and system for unmanned aerial vehicles (UAVs), the main purpose of which is to realize UI point-clicking and multi-screen simultaneous display.

[0006] To achieve the above objectives, the present invention provides a dual-screen remote control data processing method for a drone, comprising:

[0007] The remote controller, the drone, and the mobile device are powered on and started respectively to obtain a starting remote controller, a starting drone, and a starting mobile device, wherein the starting remote controller includes a main control module, a WiFi module, a USB data service communication module, an IP proxy communication module, a shared memory processing module, a liquid crystal display, an input module, and a data processor;

[0008] After pairing the WiFi module in the starting remote control with the WiFi module in the starting drone, establishing a wireless communication link between the starting remote control and the starting drone, and establishing a data connection relationship between the starting remote control and the starting drone based on the wireless communication link;

[0009] Initiating a data collection request from the start remote control to the start drone via the input module and the IP proxy communication module according to the data connection relationship, and receiving response data information from the start drone regarding the data collection request in the IP proxy communication module;

[0010] After forwarding the response data information to the data processor through the IP proxy communication module, extracting the video data and the drone status from the response data information using the data processor, and displaying the video data and the drone status on the liquid crystal display screen, thereby completing the first data processing of the starting remote controller on the starting drone and obtaining a first data processing result;

[0011] After forwarding the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module, the response data information is sent to the starting mobile device using the main control module and the USB data service communication module;

[0012] Using the USB data service communication module and the shared memory processing module, the data control information of the starting mobile device regarding the response data information is returned to the IP proxy communication module, and based on the data control information, a data control request is initiated from the starting remote controller to the starting drone via the IP proxy communication module;

[0013] Based on the data control request, the second data processing between the start remote control and the start mobile device is completed to obtain a second data processing result, and the first data processing result and the second data processing result are used as the final data processing result of the start remote control.

[0014] Optionally, the step of initiating a data collection request from the starting remote controller to the starting drone through the input module and the IP proxy communication module according to the data connection relationship includes:

[0015] Determine the control instruction corresponding to the IP proxy communication module through the input module;

[0016] Obtaining a control command proxy forwarding module and a remote control serial port forwarding proxy module in the IP proxy communication module;

[0017] forwarding the control instruction from the input module to the IP proxy communication module using the remote controller serial port forwarding proxy module;

[0018] According to the data connection relationship, the control instruction proxy forwarding module is used to send the control instruction from the IP proxy communication module to the starting drone, so as to complete the process of initiating a data collection request from the starting remote controller to the starting drone.

[0019] Optionally, forwarding the response data information to the data processor through the IP proxy communication module includes:

[0020] Obtain a video stream proxy forwarding module and an FTP file proxy forwarding module in the IP proxy communication module;

[0021] Identifying the video stream information and the file information in the response data information;

[0022] forwarding the video stream information to the data processor using the video stream proxy forwarding module;

[0023] The FTP file proxy forwarding module is utilized to forward the file information to the data processor.

[0024] Optionally, forwarding the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module includes:

[0025] Determining a buffer corresponding to the response data information in a shared memory of the startup remote controller;

[0026] Querying the sending memory area corresponding to the response data information and the sending waiting queue of the sending memory area in the buffer;

[0027] Transmitting the response data information to the shared memory processing module through the IP proxy communication module;

[0028] Using the sending buffer operation in the shared memory processing module to add the response data information to the sending waiting queue;

[0029] Utilize the mutual exclusion operation in the shared memory processing module to seize the free memory area in the sending memory area;

[0030] After waking up the response data information in the sending waiting queue by using the wake-up operation in the shared memory processing module, the response data information is stored in an idle memory area in the sending memory area to obtain an information memory area;

[0031] Release the information memory area in the sending memory area by using the release operation in the shared memory processing module;

[0032] The USB data service communication module queries the response data information in the information memory area to complete the process of forwarding the response data information to the USB data service communication module.

[0033] Optionally, the using the main control module and the USB data service communication module to send the response data information to the starting mobile device includes:

[0034] Using the main control module to monitor the device status of the activated mobile device;

[0035] Determine whether the device is in an inserted state;

[0036] When the device state is the inserted state, using the main control module to identify the device type of the mobile device to be started;

[0037] Based on the device type, the main control module and the USB data service communication module are used to send the response data information to the starting mobile device.

[0038] Optionally, based on the device type, using the main control module and the USB data service communication module to send the response data information to the starting mobile device includes:

[0039] When the device type is an iOS device type, pairing the USB data service communication module with the starting mobile device using the main control module to obtain a pairing relationship between the USB data service communication module and the starting mobile device;

[0040] According to the pairing relationship, query the ios unit in the USB data service communication module;

[0041] Using the ios unit to perform packet processing on the response data information to obtain an ios sealed data packet;

[0042] Using host communication technology between the iOS unit and the starting mobile device to transmit the iOS sealed data packet to the data parsing process in the starting mobile device, thereby completing the process of sending the response data information to the starting mobile device;

[0043] When the device type is an Android device type, switching the current mode of the USB data service communication module to the AOA mode;

[0044] According to the AOA mode, query the Android unit in the USB data service communication module;

[0045] Using the Android unit to perform packet processing on the response data information to obtain an Android sealed data packet;

[0046] The asynchronous communication technology between the Android unit and the starting mobile device is used to transmit the Android sealed data packet to the data parsing process in the starting mobile device to complete the process of sending the response data information to the starting mobile device.

[0047] Optionally, the using the USB data service communication module and the shared memory processing module to return the data control information of the activated mobile device regarding the response data information to the IP proxy communication module includes:

[0048] After receiving the data control information sent by the starting mobile device in the USB data service communication module, the data control information is unpacked according to the USB communication protocol corresponding to the data control information to obtain an unpacked data packet;

[0049] Determining a buffer corresponding to the decapsulated data packet in a shared memory of the startup remote controller;

[0050] Querying the receiving memory area corresponding to the decapsulated data packet and the receiving waiting queue of the receiving memory area in the buffer;

[0051] The USB data service communication module transmits the decapsulated data packet to the shared memory processing module;

[0052] Adding the decapsulated data packet to the receiving waiting queue by utilizing the receiving buffer operation in the shared memory processing module;

[0053] Utilize the mutual exclusion operation in the shared memory processing module to seize the free memory area in the receiving memory area;

[0054] After waking up the decapsulated data packets in the receiving waiting queue by using the wake-up operation in the shared memory processing module, the decapsulated data packets are stored in an idle memory area in the receiving memory area to obtain a data memory area;

[0055] Release the data memory area in the sending memory area by using the release operation in the shared memory processing module;

[0056] The decapsulated data packet in the data memory area is queried by the IP proxy communication module to complete the process of returning the data control information of the starting mobile device regarding the response data information to the IP proxy communication module.

[0057] Optionally, the initiating a data control request from the starting remote controller to the starting drone through the IP proxy communication module based on the data control information includes:

[0058] Obtaining a control instruction proxy forwarding module in the IP proxy communication module;

[0059] The control instruction proxy forwarding module is used to forward the data control information from the starting remote controller to the starting drone, so as to complete the process of initiating a data control request from the starting remote controller to the starting drone.

[0060] Optionally, completing the second data processing between the starting remote controller and the starting mobile device based on the data control request to obtain a second data processing result includes:

[0061] Based on the data control request, data collection and control are performed on the started UAV to obtain the UAV in the control state;

[0062] Collecting real-time data packets of the UAV under the control state;

[0063] Sending the real-time data packet to the start remote controller;

[0064] Performing first data display on the real-time data packet on the starting remote controller to obtain a first data display result;

[0065] Sending a real-time data packet to the starting mobile device via the starting remote controller;

[0066] Performing second data display on the real-time data packet on the starting mobile device to obtain a second data display result;

[0067] According to the first data display result and the second data display result, the second data processing between the starting remote controller and the starting mobile device is completed to obtain a second data processing result.

[0068] In order to solve the above problems, the present invention also provides a dual-screen remote control data processing system for a drone, the system comprising:

[0069] The device startup module is used to power on the remote control, the drone, and the mobile device respectively to obtain the startup remote control, the startup drone, and the startup mobile device, wherein the startup remote control includes a main control module, a WiFi module, a USB data service communication module, an IP proxy communication module, a shared memory processing module, a liquid crystal display, an input module, and a data processor;

[0070] a connection establishment module, configured to establish a wireless communication link between the remote controller and the drone after pairing the WiFi module in the remote controller and the drone, and to establish a data connection between the remote controller and the drone based on the wireless communication link;

[0071] an information receiving module, configured to initiate a data collection request from the starting remote controller to the starting drone via the input module and the IP proxy communication module according to the data connection relationship, and receive, in the IP proxy communication module, response data information from the starting drone regarding the data collection request;

[0072] a data display module, configured to, after forwarding the response data information to the data processor via the IP proxy communication module, extract the video data and the drone status from the response data information using the data processor, and display the video data and the drone status on the liquid crystal display screen, thereby completing a first data processing of the start remote controller on the start drone and obtaining a first data processing result;

[0073] an information sending module, configured to forward the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module, and then send the response data information to the starting mobile device using the main control module and the USB data service communication module;

[0074] a request initiating module, configured to use the USB data service communication module and the shared memory processing module to return data control information of the starting mobile device regarding the response data information to the IP proxy communication module, and based on the data control information, initiate a data control request from the starting remote controller to the starting drone via the IP proxy communication module;

[0075] The result determination module is used to complete the second data processing between the start remote control and the start mobile device based on the data control request, obtain a second data processing result, and use the first data processing result and the second data processing result as the final data processing result of the start remote control.

[0076] Compared with the problems described in the background technology, the present invention extends a USB interface on the remote control to link to the user's existing mobile phone screen, thereby realizing UI and other interactive capabilities. This interactive capability includes using the mobile data of the mobile phone to realize cloud interaction, using the GPS information of the mobile phone to realize positioning tracking interaction, etc., and has the ability of dual-screen display to enhance user experience. This solution can solve the problem of no UI interactive capability in the case of existing low-cost non-touch display screens, and can realize the simultaneous display of data on multiple screens. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a flow chart of a dual-screen remote controller data processing method for a drone provided by one embodiment of the present invention;

[0078] Figure 2 A schematic diagram illustrating a method for processing data of a dual-screen remote controller for a drone according to an embodiment of the present invention;

[0079] Figure 3 A schematic diagram of a module for implementing the dual-screen remote control data processing method for a drone, provided in one embodiment of the present invention.

[0080] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0081] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0082] The embodiments of the present application provide a method for processing data of a dual-screen remote controller for use with a drone. The execution subject of the dual-screen remote controller data processing method for use with a drone includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiments of the present application. In other words, the dual-screen remote controller data processing method for use with a drone can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0083] Reference Figure 1 FIG2 is a flow chart of a dual-screen remote control data processing method for a drone provided by an embodiment of the present invention. In this embodiment, the dual-screen remote control data processing method for a drone includes:

[0084] S1. Power on the remote control, the drone, and the mobile device respectively to obtain a starting remote control, a starting drone, and a starting mobile device, wherein the starting remote control includes a main control module, a WiFi module, a USB data service communication module, an IP proxy communication module, a shared memory processing module, a liquid crystal display, an input module, and a data processor.

[0085] Among them, the mobile device refers to a mobile phone device, the main control module is responsible for monitoring the status of the USB device (the USB device is the mobile device), identifying the type of USB device, and performing AOA switching and IOS pairing for different USB device types, preparing all preliminary work matters for USB communication, and notifying other modules. The WiFi module is used to pair with the WiFi module in the drone, and the USB data service communication module is used to receive data sent from the USB device end, unpack the data according to the USB communication protocol, and notify the IP proxy communication module to process the unpacked data through BUFFER sharing, and at the same time receive the data packet returned by the IP proxy communication module, encapsulate it and send it to the USB device end. The USB data service communication module includes the iOS USB data service communication module and the Android USB data service communication module. The iOS USB data service communication module communicates with the USB device through UNDX SOCKET, and the Android The USB data service communication module communicates with USB devices based on the asynchronous USB communication in the BULK mode of the libusb library block. The IP proxy communication module contains a control instruction proxy forwarding module, a video stream proxy forwarding module, and an FTP file proxy forwarding module. The IP proxy communication module is the core module of the remote control. It receives data sent by the USB data service communication module through BUFFER sharing. It is responsible for initiating an IP connection with the aircraft end and relaying data. It does not process the USB communication protocol. All USB packet unpacking and packaging are performed in the USB data service communication module. The shared memory processing module includes mutually exclusive sharing of shared memory through receiving data operations, sending data operations, and synchronous mutual exclusion operations. The receiving data operation corresponds to RX_BUFFER_LIST (the queue waiting for memory corresponding to the receiving memory area) and TX_BUFFER_LIST (the queue waiting for memory corresponding to the sending memory area). ) and MUTEX / NOTIFY, RX means receiving data, TX means sending data, MUTEX means mutual exclusion lock, which is used to lock the shared resource when accessing the shared resource to prevent other processes from disturbing the access of the current process, and release the mutex lock after the current process ends the access, that is, the mutex lock includes the operations of grabbing resources and releasing resources, NOTIFY means wake-up operation, for example, notify_one only wakes up the first thread in the queue, notify_all means that all threads are woken up one by one, and the first one to grab the lock wakes up first. It should be noted that all the RX memory areas from the USB device end to the aircraft end are used in the same group, and the TX memory areas from the aircraft end to the USB device end are used in the same group. The purpose of this design is that data does not need to be copied and the transfer efficiency is high, but it is necessary to consider using MUTEX to synchronize mutual exclusive access and queue waiting for processing of the memory area. For example, A puts data into the shared resource and then releases the shared resource.B searches the shared resource for the data put in by A. That is, the process where A executes first and B executes later is a synchronous process. The input module is a hardware input device, such as a keyboard, used to receive user input data. The data processor is used to decompress the compressed data such as the video and aircraft status sent by the aircraft, and parse the video and aircraft status contained in the data to display it on the screen.

[0086] See Figure 2 FIG. 1 is a diagram showing a relationship between a method for processing data of a dual-screen remote controller for a drone according to an embodiment of the present invention. Figure 2 In the figure, MCU1 includes a main control module, a USB data service communication module, an IP proxy communication module, a shared memory processing module and a data processor. The RS233 serial port is connected to the remote control serial port forwarding proxy module in the IP proxy communication module. MCU2 represents the data processor of the joystick and buttons.

[0087] S2. After pairing the WiFi module in the starting remote control with the WiFi module in the starting drone, establish a wireless communication link between the starting remote control and the starting drone, and establish a data connection relationship between the starting remote control and the starting drone based on the wireless communication link.

[0088] Optionally, the process of establishing a data connection relationship between the starting remote control and the starting drone based on the wireless communication link refers to the process of initiating a TCP / IP connection request through a WiFi wireless link and receiving a response request sent by the aircraft, wherein the data connection relationship refers to the relationship of establishing a TCP / IP connection between the starting remote control and the starting drone.

[0089] S3. According to the data connection relationship, a data collection request is initiated from the starting remote control to the starting drone through the input module and the IP proxy communication module, and response data information of the starting drone regarding the data collection request is received in the IP proxy communication module.

[0090] In one embodiment of the present invention, the data collection request is initiated from the starting remote control to the starting drone through the input module and the IP proxy communication module based on the data connection relationship, including: determining the control instruction corresponding to the IP proxy communication module through the input module; obtaining the control instruction proxy forwarding module and the remote control serial port forwarding proxy module in the IP proxy communication module; using the remote control serial port forwarding proxy module to forward the control instruction from the input module to the IP proxy communication module; and according to the data connection relationship, using the control instruction proxy forwarding module to send the control instruction from the IP proxy communication module to the starting drone, so as to complete the process of initiating a data collection request from the starting remote control to the starting drone.

[0091] S4. After forwarding the response data information to the data processor through the IP proxy communication module, the data processor is used to extract the video data and the drone status in the response data information, and display the video data and the drone status on the LCD screen to complete the first data processing of the start remote control on the start drone and obtain a first data processing result.

[0092] In one embodiment of the present invention, forwarding the response data information to the data processor through the IP proxy communication module includes: obtaining a video stream proxy forwarding module and an FTP file proxy forwarding module in the IP proxy communication module; identifying the video stream information and file information in the response data information; using the video stream proxy forwarding module to forward the video stream information to the data processor; and using the FTP file proxy forwarding module to forward the file information to the data processor.

[0093] S5. After forwarding the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module, the response data information is sent to the starting mobile device using the main control module and the USB data service communication module.

[0094] In one embodiment of the present invention, forwarding the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module includes: determining a buffer corresponding to the response data information in the shared memory of the start-up remote control; querying the sending memory area corresponding to the response data information and the sending waiting queue of the sending memory area in the buffer; transmitting the response data information to the shared memory processing module through the IP proxy communication module; adding the response data information to the sending waiting queue by using a sending buffer operation in the shared memory processing module; seizing an idle memory area in the sending memory area by using a mutual exclusion operation in the shared memory processing module; waking up the response data information in the sending waiting queue by using a wake-up operation in the shared memory processing module, and then storing the response data information in an idle memory area in the sending memory area to obtain an information memory area; releasing the information memory area in the sending memory area by using a release operation in the shared memory processing module; and querying the response data information in the information memory area through the USB data service communication module to complete the process of forwarding the response data information to the USB data service communication module.

[0095] The information memory area refers to an area for storing the response data information.

[0096] In one embodiment of the present invention, the use of the main control module and the USB data service communication module to send the response data information to the starting mobile device includes: using the main control module to monitor the device status of the starting mobile device; determining whether the device status is in an inserted state; when the device status is in an inserted state, using the main control module to identify the device type of the starting mobile device; based on the device type, using the main control module and the USB data service communication module to send the response data information to the starting mobile device.

[0097] The inserted state refers to whether the data cable of the USB device is inserted into the USB interface of the remote control.

[0098] In another embodiment of the present invention, based on the device type, the main control module and the USB data service communication module are used to send the response data information to the startup mobile device, including: when the device type is an iOS device type, the main control module is used to pair the USB data service communication module with the startup mobile device to obtain a pairing relationship between the USB data service communication module and the startup mobile device; according to the pairing relationship, the iOS unit in the USB data service communication module is queried; the iOS unit is used to package the response data information to obtain an iOS sealed data packet; and the iOS sealed data packet is transmitted to the startup mobile device using the host communication technology between the iOS unit and the startup mobile device. A data parsing process in a mobile device is used to complete the process of sending the response data information to the started mobile device; when the device type is an Android device type, the current mode of the USB data service communication module is switched to the AOA mode; according to the AOA mode, the Android unit in the USB data service communication module is queried; the response data information is packaged using the Android unit to obtain an Android sealed data packet; and the Android sealed data packet is transmitted to the data parsing process in the started mobile device using asynchronous communication technology between the Android unit and the started mobile device to complete the process of sending the response data information to the started mobile device.

[0099] Among them, host communication technology refers to UNDX SOCKET, data parsing process refers to the process used to parse sealed data packets in mobile devices, asynchronous communication technology refers to asynchronous USB communication based on libusb library block BULK mode, and AOA mode refers to communicating with external USB devices via USB. In AOA mode, the Android device acts as a USB host and the external USB device acts as a USB slave. This mode is mainly used for control and data transmission between Android devices and external USB devices, such as connecting car audio, game controllers, etc.

[0100] S6. Utilize the USB data service communication module and the shared memory processing module to return the data control information of the starting mobile device regarding the response data information to the IP proxy communication module. Based on the data control information, initiate a data control request from the starting remote control to the starting drone through the IP proxy communication module.

[0101] In one embodiment of the present invention, the method of using the USB data service communication module and the shared memory processing module to return the data control information of the starting mobile device regarding the response data information to the IP proxy communication module includes: after receiving the data control information sent by the starting mobile device in the USB data service communication module, unpacking the data control information according to the USB communication protocol corresponding to the data control information to obtain an unpacked data packet; determining a buffer corresponding to the unpacked data packet in the shared memory of the starting remote controller; querying the receiving memory area corresponding to the unpacked data packet and the receiving waiting queue of the receiving memory area in the buffer; transmitting the unpacked data packet to the shared memory through the USB data service communication module. processing module; using the receiving buffer operation in the shared memory processing module to add the unsealed data packet to the receiving waiting queue; using the mutual exclusion operation in the shared memory processing module to seize the free memory area in the receiving memory area; using the wake-up operation in the shared memory processing module to wake up the unsealed data packet in the receiving waiting queue, and then storing the unsealed data packet in the free memory area in the receiving memory area to obtain a data memory area; using the release operation in the shared memory processing module to release the data memory area in the sending memory area; querying the unsealed data packet in the data memory area through the IP proxy communication module to complete the process of returning the data control information of the starting mobile device about the response data information to the IP proxy communication module.

[0102] Among them, the data control information refers to the drone control information entered by the user on the mobile device, the USB communication protocol refers to the protocol used for communication between the USB device and the remote control, the buffer is consistent with the buffer corresponding to the aforementioned response data information, and the data memory area refers to the area for storing the unpacked data packet.

[0103] In one embodiment of the present invention, based on the data control information, a data control request is initiated from the starting remote control to the starting drone through the IP proxy communication module, including: obtaining a control instruction proxy forwarding module in the IP proxy communication module; using the control instruction proxy forwarding module to forward the data control information from the starting remote control to the starting drone, so as to complete the process of initiating a data control request from the starting remote control to the starting drone.

[0104] S7. Based on the data control request, complete the second data processing between the start remote control and the start mobile device, obtain a second data processing result, and use the first data processing result and the second data processing result as the final data processing result of the start remote control.

[0105] In one embodiment of the present invention, based on the data control request, the second data processing between the starting remote control and the starting mobile device is completed to obtain a second data processing result, including: based on the data control request, data collection and control of the starting drone is performed to obtain a controlled drone; real-time data packets under the controlled drone are collected; the real-time data packets are sent to the starting remote control; first data display of the real-time data packets is performed on the starting remote control to obtain a first data display result; the real-time data packets are sent to the starting mobile device via the starting remote control; second data display of the real-time data packets is performed on the starting mobile device to obtain a second data display result; according to the first data display result and the second data display result, the second data processing between the starting remote control and the starting mobile device is completed to obtain a second data processing result.

[0106] The controlled drone refers to a drone that performs video acquisition after receiving a user control signal, and the real-time data packet includes the video acquired by the drone and its own state.

[0107] Compared with the problems described in the background technology, the present invention extends a USB interface on the remote control to link to the user's existing mobile phone screen, thereby realizing UI and other interactive capabilities. This interactive capability includes using the mobile data of the mobile phone to realize cloud interaction, using the GPS information of the mobile phone to realize positioning tracking interaction, etc., and has the ability of dual-screen display to enhance user experience. This solution can solve the problem of no UI interactive capability in the case of existing low-cost non-touch display screens, and can realize the simultaneous display of data on multiple screens.

[0108] like Figure 3 FIG. 1 is a functional module diagram of a dual-screen remote controller data processing system for a drone according to the present invention.

[0109] The dual-screen remote control data processing system 300 for drones described in the present invention can be installed in an electronic device. Depending on the functionality implemented, the dual-screen remote control data processing system for drones can include a device startup module 301, a connection establishment module 302, an information reception module 303, a data display module 304, an information transmission module 305, a request initiation module 306, and a result determination module 307. A module, also referred to as a unit, is a series of computer program segments that can be executed by an electronic device processor and perform a fixed function. These modules are stored in the electronic device's memory.

[0110] In the embodiment of the present invention, the functions of each module / unit are as follows:

[0111] The device startup module 301 is used to power on the remote control, the drone, and the mobile device respectively to obtain the startup remote control, the startup drone, and the startup mobile device. The startup remote control includes a main control module, a WiFi module, a USB data service communication module, an IP proxy communication module, a shared memory processing module, a liquid crystal display, an input module, and a data processor;

[0112] The connection establishment module 302 is configured to establish a wireless communication link between the remote controller and the drone after pairing the WiFi module in the remote controller and the drone, and to establish a data connection between the remote controller and the drone based on the wireless communication link;

[0113] The information receiving module 303 is configured to initiate a data collection request from the starting remote controller to the starting drone via the input module and the IP proxy communication module according to the data connection relationship, and receive response data information from the starting drone regarding the data collection request in the IP proxy communication module;

[0114] The data display module 304 is configured to forward the response data information to the data processor via the IP proxy communication module, extract the video data and the drone status from the response data information using the data processor, and display the video data and the drone status on the LCD screen, thereby completing the first data processing of the remote controller on the started drone and obtaining a first data processing result.

[0115] The information sending module 305 is configured to forward the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module, and then send the response data information to the starting mobile device using the main control module and the USB data service communication module;

[0116] The request initiating module 306 is configured to use the USB data service communication module and the shared memory processing module to return data control information of the starting mobile device regarding the response data information to the IP proxy communication module, and based on the data control information, initiate a data control request from the starting remote controller to the starting drone via the IP proxy communication module;

[0117] The result determination module 307 is used to complete the second data processing between the start remote control and the start mobile device based on the data control request, obtain a second data processing result, and use the first data processing result and the second data processing result as the final data processing result of the start remote control.

[0118] In detail, the modules in the dual-screen remote controller data processing system 300 for drones according to the embodiment of the present invention are used in the same manner as above. Figure 1 The same technical means as the dual-screen remote control data processing method applied to drones described in , and can produce the same technical effects, will not be repeated here.

[0119] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dual-screen remote control data processing method for drones, characterized in that: The method comprises: The remote controller, the drone, and the mobile device are powered on and started respectively to obtain a starting remote controller, a starting drone, and a starting mobile device, wherein the starting remote controller includes a main control module, a WiFi module, a USB data service communication module, an IP proxy communication module, a shared memory processing module, a liquid crystal display, an input module, and a data processor; After pairing the WiFi module in the starting remote control with the WiFi module in the starting drone, establishing a wireless communication link between the starting remote control and the starting drone, and establishing a data connection relationship between the starting remote control and the starting drone based on the wireless communication link; Initiating a data collection request from the start remote control to the start drone via the input module and the IP proxy communication module according to the data connection relationship, and receiving response data information from the start drone regarding the data collection request in the IP proxy communication module; After forwarding the response data information to the data processor through the IP proxy communication module, extracting the video data and the drone status from the response data information using the data processor, and displaying the video data and the drone status on the liquid crystal display screen, thereby completing the first data processing of the starting remote controller on the starting drone and obtaining a first data processing result; After forwarding the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module, the response data information is sent to the starting mobile device by using the main control module and the USB data service communication module, wherein forwarding the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module includes: Determining a buffer corresponding to the response data information in a shared memory of the startup remote controller; Querying the sending memory area corresponding to the response data information and the sending waiting queue of the sending memory area in the buffer; Transmitting the response data information to the shared memory processing module through the IP proxy communication module; Using the sending buffer operation in the shared memory processing module to add the response data information to the sending waiting queue; Utilize the mutual exclusion operation in the shared memory processing module to seize the free memory area in the sending memory area; After waking up the response data information in the sending waiting queue by using the wake-up operation in the shared memory processing module, the response data information is stored in an idle memory area in the sending memory area to obtain an information memory area; Release the information memory area in the sending memory area by using the release operation in the shared memory processing module; querying the response data information in the information memory area through the USB data service communication module to complete the process of forwarding the response data information to the USB data service communication module; Using the USB data service communication module and the shared memory processing module, the data control information of the starting mobile device regarding the response data information is returned to the IP proxy communication module, and based on the data control information, a data control request is initiated from the starting remote controller to the starting drone via the IP proxy communication module; Based on the data control request, the second data processing between the start remote control and the start mobile device is completed to obtain a second data processing result, and the first data processing result and the second data processing result are used as the final data processing result of the start remote control.

2. The dual-screen remote controller data processing method for a drone as claimed in claim 1, wherein: The method of initiating a data collection request from the starting remote controller to the starting drone through the input module and the IP proxy communication module according to the data connection relationship includes: Determine the control instruction corresponding to the IP proxy communication module through the input module; Obtaining a control command proxy forwarding module and a remote control serial port forwarding proxy module in the IP proxy communication module; forwarding the control instruction from the input module to the IP proxy communication module using the remote controller serial port forwarding proxy module; According to the data connection relationship, the control instruction proxy forwarding module is used to send the control instruction from the IP proxy communication module to the starting drone, so as to complete the process of initiating a data collection request from the starting remote controller to the starting drone.

3. The dual-screen remote controller data processing method for a drone as claimed in claim 1, wherein: The step of forwarding the response data information to the data processor through the IP proxy communication module includes: Obtain a video stream proxy forwarding module and an FTP file proxy forwarding module in the IP proxy communication module; Identifying the video stream information and the file information in the response data information; forwarding the video stream information to the data processor using the video stream proxy forwarding module; The FTP file proxy forwarding module is utilized to forward the file information to the data processor.

4. The dual-screen remote controller data processing method for a drone as claimed in claim 1, wherein: The step of using the main control module and the USB data service communication module to send the response data information to the starting mobile device includes: Using the main control module to monitor the device status of the activated mobile device; Determine whether the device is in an inserted state; When the device state is the inserted state, using the main control module to identify the device type of the mobile device to be started; Based on the device type, the main control module and the USB data service communication module are used to send the response data information to the starting mobile device.

5. The dual-screen remote controller data processing method for a drone as claimed in claim 4, wherein: The sending of the response data information to the starting mobile device by using the main control module and the USB data service communication module based on the device type includes: When the device type is an iOS device type, pairing the USB data service communication module with the starting mobile device using the main control module to obtain a pairing relationship between the USB data service communication module and the starting mobile device; According to the pairing relationship, query the ios unit in the USB data service communication module; Using the iOS unit to perform packet processing on the response data information to obtain an iOS sealed data packet; Using host communication technology between the iOS unit and the starting mobile device to transmit the iOS sealed data packet to the data parsing process in the starting mobile device, thereby completing the process of sending the response data information to the starting mobile device; When the device type is an Android device type, switching the current mode of the USB data service communication module to the AOA mode; According to the AOA mode, query the Android unit in the USB data service communication module; Using the Android unit to perform packet processing on the response data information to obtain an Android sealed data packet; The asynchronous communication technology between the Android unit and the starting mobile device is used to transmit the Android sealed data packet to the data parsing process in the starting mobile device to complete the process of sending the response data information to the starting mobile device.

6. The dual-screen remote controller data processing method for a drone as claimed in claim 1, wherein: The method of using the USB data service communication module and the shared memory processing module to return the data control information of the mobile device regarding the response data information to the IP proxy communication module includes: After receiving the data control information sent by the starting mobile device in the USB data service communication module, the data control information is unpacked according to the USB communication protocol corresponding to the data control information to obtain an unpacked data packet; Determining a buffer corresponding to the decapsulated data packet in a shared memory of the startup remote controller; Querying the receiving memory area corresponding to the decapsulated data packet and the receiving waiting queue of the receiving memory area in the buffer; The USB data service communication module transmits the decapsulated data packet to the shared memory processing module; Adding the decapsulated data packet to the receiving waiting queue by utilizing the receiving buffer operation in the shared memory processing module; Utilize the mutual exclusion operation in the shared memory processing module to seize the free memory area in the receiving memory area; After waking up the decapsulated data packets in the receiving waiting queue by using the wake-up operation in the shared memory processing module, the decapsulated data packets are stored in an idle memory area in the receiving memory area to obtain a data memory area; Release the data memory area in the sending memory area by using the release operation in the shared memory processing module; The decapsulated data packet in the data memory area is queried by the IP proxy communication module to complete the process of returning the data control information of the starting mobile device regarding the response data information to the IP proxy communication module.

7. The dual-screen remote controller data processing method for a drone as claimed in claim 1, wherein: The initiating a data control request from the starting remote controller to the starting drone through the IP proxy communication module based on the data control information includes: Obtaining a control instruction proxy forwarding module in the IP proxy communication module; The control instruction proxy forwarding module is used to forward the data control information from the starting remote controller to the starting drone, so as to complete the process of initiating a data control request from the starting remote controller to the starting drone.

8. The dual-screen remote controller data processing method for a drone as claimed in claim 1, wherein: The step of completing the second data processing between the starting remote controller and the starting mobile device based on the data control request to obtain a second data processing result includes: Based on the data control request, data collection and control are performed on the started UAV to obtain the UAV in the control state; Collecting real-time data packets of the UAV under the control state; Sending the real-time data packet to the start remote controller; Performing first data display on the real-time data packet on the starting remote controller to obtain a first data display result; Sending a real-time data packet to the starting mobile device via the starting remote controller; Performing second data display on the real-time data packet on the starting mobile device to obtain a second data display result; According to the first data display result and the second data display result, the second data processing between the starting remote controller and the starting mobile device is completed to obtain a second data processing result.

9. A dual-screen remote control data processing system for drones, characterized in that: The system comprises: The device startup module is used to power on the remote control, the drone, and the mobile device respectively to obtain the startup remote control, the startup drone, and the startup mobile device, wherein the startup remote control includes a main control module, a WiFi module, a USB data service communication module, an IP proxy communication module, a shared memory processing module, a liquid crystal display, an input module, and a data processor; a connection establishment module, configured to establish a wireless communication link between the remote controller and the drone after pairing the WiFi module in the remote controller and the drone, and to establish a data connection between the remote controller and the drone based on the wireless communication link; an information receiving module, configured to initiate a data collection request from the starting remote controller to the starting drone via the input module and the IP proxy communication module according to the data connection relationship, and receive, in the IP proxy communication module, response data information from the starting drone regarding the data collection request; a data display module, configured to, after forwarding the response data information to the data processor via the IP proxy communication module, extract the video data and the drone status from the response data information using the data processor, and display the video data and the drone status on the liquid crystal display screen, thereby completing a first data processing of the start remote controller on the start drone and obtaining a first data processing result; An information sending module is configured to forward the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module, and then use the main control module and the USB data service communication module to send the response data information to the starting mobile device, wherein the forwarding of the response data information to the USB data service communication module through the IP proxy communication module and the shared memory processing module includes: Determining a buffer corresponding to the response data information in a shared memory of the startup remote controller; Querying the sending memory area corresponding to the response data information and the sending waiting queue of the sending memory area in the buffer; Transmitting the response data information to the shared memory processing module through the IP proxy communication module; Using the sending buffer operation in the shared memory processing module to add the response data information to the sending waiting queue; Utilize the mutual exclusion operation in the shared memory processing module to seize the free memory area in the sending memory area; After waking up the response data information in the sending waiting queue by using the wake-up operation in the shared memory processing module, the response data information is stored in an idle memory area in the sending memory area to obtain an information memory area; Release the information memory area in the sending memory area by using the release operation in the shared memory processing module; querying the response data information in the information memory area through the USB data service communication module to complete the process of forwarding the response data information to the USB data service communication module; a request initiating module, configured to use the USB data service communication module and the shared memory processing module to return data control information of the starting mobile device regarding the response data information to the IP proxy communication module, and based on the data control information, initiate a data control request from the starting remote controller to the starting drone via the IP proxy communication module; The result determination module is used to complete the second data processing between the start remote control and the start mobile device based on the data control request, obtain a second data processing result, and use the first data processing result and the second data processing result as the final data processing result of the start remote control.

Images