Unmanned aerial vehicle ground station control device and method based on multi-mode somatosensory interaction and virtual reality and unmanned aerial vehicle system

Abstract

The invention provides an unmanned aerial vehicle ground station control device and method based on multi-mode somatosensory interaction and virtual reality and an unmanned aerial vehicle system. The device comprises a main control module, a multi-mode interaction module used for collecting eye images and hand images, and a VR display module. According to the unmanned aerial vehicle ground station control device, a multi-mode somatosensory interaction fusion mechanism is introduced, so that more accurate and safer unmanned aerial vehicle control is realized.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV ground station control device, method, and UAV system based on multimodal haptic interaction and virtual reality. Background Technology

[0002] Currently, most improvements to ground stations are made using a combination of technologies, including virtual reality. An improved ground station typically consists of the following components: a communication module for data transmission with the UAV itself; an image control module for processing data transmitted from the UAV's image module; a control system for processing operator input commands; and corresponding modules such as a storage module, an aircraft adaptation module, and an information processing module.

[0003] However, the aforementioned ground station systems still use traditional PCs, which have the following drawbacks: they rely on a computer host and an external display screen, making them difficult to deploy and carry quickly; most systems still use a handheld controller or touchscreen as the main input device, occupying the operator's hands and preventing natural and intuitive three-dimensional spatial control; the complex menu operation steps are cumbersome and easily distract attention from the flight situation; the operator's eyes must simultaneously bear the high-intensity task of receiving video information and sending control commands, leading to visual fatigue and decreased control accuracy. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a UAV ground station control device, method and UAV system based on multimodal haptic interaction and virtual reality, so as to solve the problem of low ground station control accuracy.

[0005] To achieve the above and other related objectives, a first aspect of this application provides a UAV ground station control device based on multimodal motion-sensing interaction and virtual reality, connected to a UAV, comprising: a main control module, a multimodal interaction module, and a VR display module; wherein the main control module is connected to both the multimodal interaction module and the VR display module; the multimodal interaction module is used to acquire eye images and hand images at the current moment; the main control module is used to receive video stream information acquired by the UAV's camera module at the current moment, and decode and re-encode the video stream information at the current moment to obtain the display data of the VR display module at the current moment; the main control module is also used to generate a camera module control signal at the current moment based on a pre-acquired adjustment signal generation rule based on eye movements, using an eye recognition algorithm, based on the eye images at the current moment, to adjust the camera module of the UAV accordingly; and, based on a pre-acquired control signal generation rule based on hand movements, using a hand movement recognition model, to generate a UAV attitude control signal at the current moment based on the hand images at the current moment, to control the attitude of the UAV accordingly.

[0006] In some embodiments of the first aspect of this application, based on pre-acquired adjustment signal generation rules based on eye movements, and using an eye recognition algorithm, a camera module control signal for the current moment is generated based on the eye image at the current moment. This includes: preprocessing the eye image at the current moment to obtain a preprocessed eye image; processing the preprocessed eye image at the current moment using the eye recognition algorithm to obtain fixation point data and orbital contour data at the current moment; comparing the fixation point data and orbital contour data at the current moment with pre-acquired eye reference data to obtain a comparison result at the current moment; and obtaining the camera module control signal at the current moment based on the comparison result at the current moment, according to the pre-acquired adjustment signal generation rules based on eye movements.

[0007] In some embodiments of the first aspect of this application, the eye recognition algorithm includes: selecting multiple pupil calculation points and multiple orbital calculation points on a preprocessed eye image at the current moment, and obtaining pupil shape data and orbital contour data at the current moment based on the multiple pupil calculation points and the multiple orbital calculation points; and obtaining gaze point data at the current moment based on the pupil shape data at the current moment using a pre-trained gaze point calculation model.

[0008] In some embodiments of the first aspect of this application, the rule for generating the adjustment signal based on eye movements includes: if the horizontal displacement of the gaze point in the comparison result exceeds a set horizontal threshold, then a horizontal movement signal of the camera module is generated; if the horizontal displacement of the gaze point in the comparison result does not exceed the set horizontal threshold but the vertical displacement of the gaze point exceeds a set vertical threshold, then a vertical movement signal of the camera module is generated; if only the change in the orbital area in the comparison result exceeds a set orbital area change threshold, then a focal length adjustment signal of the camera module is generated.

[0009] In some embodiments of the first aspect of this application, based on pre-acquired control signal generation rules based on hand gestures, a hand gesture recognition model is used to generate a drone attitude control signal for the current moment based on a hand image at the current moment, including: inputting the hand image at the current moment into a pre-trained hand gesture recognition model to obtain a gesture recognition result at the current moment; and obtaining the drone attitude control signal for the current moment based on the pre-acquired control signal generation rules based on the gesture recognition result at the current moment.

[0010] In some embodiments of the first aspect of this application, the method of training a hand gesture recognition model includes: acquiring a hand gesture training set and annotating each image in the hand gesture training set with gestures; using the annotated hand gesture training set and with the assistance of a normalized attention module, training an attention-based gesture recognition network to obtain a hand gesture recognition model; wherein the attention-based gesture recognition network includes: a feature extraction layer, an attention module, two fully connected layers, and an output layer connected in sequence; the output layer is also connected to an object detection network.

[0011] In some embodiments of the first aspect of this application, the control signal generation rule based on hand gestures includes: if the gesture recognition result is that the thumb is active, then a drone ascent speed control signal is generated; if the gesture recognition result is that the thumb is not active and the index finger is active, then a drone forward speed control signal is generated; if the gesture recognition result is that only the wrist is active, then a drone rightward speed control signal is generated.

[0012] In some embodiments of the first aspect of this application, the multimodal interaction module includes: an eye-tracking unit for acquiring eye images and a hand motion capture unit for acquiring hand images; the multimodal interaction module further includes: a face capture unit; wherein the face capture unit is used to acquire a face image at the current moment and send it to the control module; the control module uses a face recognition model based on the face image at the current moment to obtain the face motion recognition result at the current moment, so as to assist in adjusting the camera module of the drone.

[0013] To achieve the above and other related objectives, a second aspect of this application provides a drone control method based on multimodal haptic interaction and virtual reality for controlling a drone, comprising: acquiring eye images and hand images at a current moment; generating a camera module control signal at a current moment based on the current eye images using an eye recognition algorithm, based on pre-acquired adjustment signal generation rules based on eye movements, to adjust the camera module of the drone accordingly; and generating a drone attitude control signal at a current moment based on the current hand images using a hand movement recognition model, based on the current hand images, to perform corresponding attitude control on the drone.

[0014] To achieve the above and other related objectives, a third aspect of this application provides an unmanned aerial vehicle (UAV) system, the UAV system comprising: a UAV ground station control device based on multimodal motion interaction and virtual reality as described above, and a UAV connected to the UAV ground station control device based on multimodal motion interaction and virtual reality.

[0015] As described above, the UAV ground station control device, method, and UAV system based on multimodal motion-sensing interaction and virtual reality of this application have the following beneficial effects: This application provides a UAV ground station control device, method, and UAV system based on multimodal motion-sensing interaction and virtual reality. The device includes a main control module, a multimodal interaction module for acquiring eye and hand images, and a VR display module. The UAV ground station control device of this application achieves more precise and safer UAV control by introducing a multimodal motion-sensing interaction fusion mechanism. Attached Figure Description

[0016] Figure 1 The diagram shown is a schematic block diagram of a UAV ground station control device based on multimodal haptic interaction and virtual reality in one embodiment of this application.

[0017] Figure 2 The diagram shown is a schematic block diagram of a multimodal interaction module in one embodiment of this application.

[0018] Figure 3 The image shown is a schematic diagram of a scene of a UAV ground station control device based on multimodal haptic interaction and virtual reality in a specific embodiment of this application.

[0019] Figure 4 The diagram shown is a schematic representation of the video stream information transmission process in one embodiment of this application.

[0020] Figure 5 The diagram shown is a schematic representation of an eye recognition process in one embodiment of this application.

[0021] Figure 6 The diagram shown is a schematic representation of the facial recognition process in one embodiment of this application.

[0022] Figure 7 The diagram shows different gestures in a specific embodiment of this application.

[0023] Figure 8 The diagram shown is a schematic representation of a gesture recognition network based on an attention mechanism in one embodiment of this application.

[0024] Figure 9 The diagram shown is a schematic representation of the training process of a gesture recognition model in one embodiment of this application.

[0025] Figure 10 The diagram shown is a flowchart illustrating the control process of a drone in one embodiment of this application.

[0026] Figure 11 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0027] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0028] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0029] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0030] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0031] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This illustration shows a schematic block diagram of a UAV ground station control device based on multimodal motion-sensing interaction and virtual reality, according to an embodiment of the present invention. The UAV ground station control device 1 based on multimodal motion-sensing interaction and virtual reality in this embodiment is connected to a UAV 2. The UAV ground station control device 1 based on multimodal motion-sensing interaction and virtual reality includes:

[0032] The main control module 11, the multimodal interaction module 12, and the VR display module 13 are connected to the multimodal interaction module 12 and the VR display module 13 respectively.

[0033] The multimodal interaction module 12 is used to acquire eye and hand images at the current moment;

[0034] The main control module 11 is used to receive the video stream information of the current moment collected by the camera module of the UAV 2, and decode and re-encode the video stream information of the current moment to obtain the display data of the VR display module 13 at the current moment;

[0035] The main control module 11 is also used to generate a camera module control signal based on the current eye image, using an eye recognition algorithm and a pre-acquired adjustment signal generation rule based on eye movement, so as to adjust the camera module of the UAV 2 accordingly; and to generate a UAV attitude control signal based on the current hand image, using a hand movement recognition model and a pre-acquired control signal generation rule based on hand movement, so as to control the attitude of the UAV 2 accordingly.

[0036] In one specific embodiment, to make the UAV ground station control device based on multimodal motion-sensing interaction and virtual reality more portable and adaptable to various scenarios requiring rapid deployment and portability, the main control module adopts an embedded host. It should be noted that an embedded host is a miniaturized, specialized computing device designed specifically for a particular scenario. It integrates core components such as processors, memory, storage, and interfaces, aiming to be embedded into a larger system or device to perform specific functions such as data processing, control, and communication, rather than being used independently as a general-purpose computer.

[0037] In one embodiment, such as Figure 2 As shown, the multimodal interaction module 12 includes an eye-tracking unit 121, a face capture unit 122, and a hand motion capture unit 123. The eye-tracking unit 121, face capture unit 122, and hand motion capture unit 123 are respectively connected to the main control module 11. The eye-tracking unit 121 is used to capture images of the user's eyes; the face capture unit 122 is used to capture images of the user's face; and the hand motion capture unit 123 is used to capture images of the user's hands.

[0038] In one embodiment, the main control module connects to the drone via a wireless communication module. The types of wireless communication modules include, but are not limited to, Bluetooth modules, Wi-Fi modules, ZigBee modules, NB-IoT modules, and eMTC modules. A suitable wireless communication module can be selected according to actual needs, and this embodiment does not limit this selection. Preferably, the main control module establishes a passthrough connection with the drone via the wireless communication module. It should be understood that a passthrough connection is a data transmission mode whose core characteristic is that it does not modify, parse, or process the transmitted data; it merely acts as a channel to transmit data unchanged from one end to the other, similar to transparent transmission at the physical layer.

[0039] In one embodiment, the type of VR (Virtual Reality) display module includes, but is not limited to, VR glasses, VR headsets, etc., which can be selected according to actual needs. This embodiment does not limit this.

[0040] In one embodiment, the VR display module and the control module are connected via an eDP interface. It should be understood that eDP (Embedded DisplayPort) is a digital video interface standard for embedded systems.

[0041] Figure 3 This illustration shows a scene diagram of a UAV ground station control device based on multimodal haptic interaction and virtual reality, according to a specific embodiment of the present invention. Figure 2 In this scenario, A represents the ground station device, and the main control module is located within the ground station device. B represents the first camera device, which is a hand motion capture unit used to capture images of the user's hands. C represents the VR glasses. The VR glasses C are equipped with a second camera device, which can be one or two, used to capture images of the user's eyes and face. It should be understood that the camera device can be a webcam or other device with camera functionality; this is not limited here.

[0042] In one embodiment, such as Figure 4As shown, after the UAV establishes a connection with the main control module of the UAV ground station control device, the UAV's camera module (e.g., a webcam) acquires video stream information in real time and transmits it back to the control module. The control module processes the compressed video stream information received by the wireless communication module to obtain a signal that can be recognized and read by the software. This process is called decoding. The signal is transmitted to the UAV control software in the form of data frames via the MAVLink v2.0 communication protocol. The data frame structure includes: packet start marker, payload length, compatibility flag, packet sequence number, system ID, component ID, message ID, payload data, and checksum data. The UAV control software re-encodes the signal to obtain display data and sends it to the VR display module for display via the eDP interface. The UAV control software modifies the user interface based on the open-source QGroundControl to adapt to VR display. It should be noted that the transmission, encoding, and decoding of the video stream can all refer to existing technologies, which will not be elaborated here. Due to the openness of MAVLink, modular sensors can be installed on other parts of the drone depending on the drone's payload. These payloads can be controlled through training and testing to expand functionality and quickly adapt to diverse application scenarios such as agricultural plant protection and power line inspection.

[0043] In one embodiment, based on pre-acquired adjustment signal generation rules based on eye movements, an eye recognition algorithm is used to generate a camera module control signal for the current moment based on the eye image at the current moment. This includes: preprocessing the eye image at the current moment to obtain a preprocessed eye image; processing the preprocessed eye image at the current moment using the eye recognition algorithm to obtain fixation point data and orbital contour data for the current moment; comparing the fixation point data and orbital contour data at the current moment with pre-acquired eye reference data to obtain a comparison result for the current moment; and obtaining the camera module control signal for the current moment based on the comparison result at the current moment, according to the pre-acquired adjustment signal generation rules based on eye movements.

[0044] In one embodiment, the eye recognition algorithm includes: selecting multiple pupil calculation points and multiple orbital calculation points on the preprocessed eye image at the current moment, and obtaining pupil shape data and orbital contour data at the current moment based on the multiple pupil calculation points and the multiple orbital calculation points; and obtaining gaze point data at the current moment based on the pupil shape data at the current moment using a pre-trained gaze point calculation model.

[0045] In one embodiment, the rule for generating adjustment signals based on eye movements includes: if the horizontal displacement of the gaze point in the comparison result exceeds a set horizontal threshold, a horizontal movement signal of the camera module is generated; if the horizontal displacement of the gaze point in the comparison result does not exceed the set horizontal threshold but the vertical displacement of the gaze point exceeds a set vertical threshold, a vertical movement signal of the camera module is generated; if only the change in the orbital area in the comparison result exceeds a set orbital area change threshold, a focal length adjustment signal of the camera module is generated.

[0046] The following will be combined with the appendix Figure 5 This section explains the specific methods used to adjust the camera module of a drone through eye recognition:

[0047] When the eye-tracking unit is activated, it continuously captures five images of the user's eyes, calculating the user's initial gaze point data and initial orbital contour data. The initial gaze point data and initial orbital contour data constitute the eye baseline data.

[0048] The eye-tracking unit acquires an image of the user's eyes at the current moment. This image is then preprocessed to obtain a preprocessed version of the eye image at that moment. The preprocessing includes image grayscale conversion, noise filtering, and image feature equalization. It should be understood that noise filtering is a technique in image processing used to remove or reduce random interference signals (noise) in an image, aiming to improve image quality and provide clearer input image features for subsequent analysis. Image equalization adjusts the distribution of image features (such as grayscale values, color channels, and texture features) to make the features more uniform within their value range, thus improving the effectiveness of subsequent processing.

[0049] Furthermore, the pupil is typically a low-grayscale region (dark area) in an image. First, pixels with grayscale values ​​below a set threshold are selected (preliminary separation of pupil candidate areas). Then, the selected areas are filled to complete any holes or breaks within the pupil, ultimately obtaining a preliminary pupil region. Simultaneously, an object detection algorithm can be used to identify the initial orbital contour. Further, multiple points are selected within the preliminary pupil region as pupil calculation points, and multiple points are selected within the initial orbital contour as orbital calculation points. In this embodiment, the number of pupil calculation points and orbital calculation points is five each; however, other numbers are possible and not limited in this embodiment.

[0050] Furthermore, based on multiple pupil calculation points, the pupil shape data at the current moment is calculated, and based on multiple orbit calculation points, the orbital contour data at the current moment is calculated. Based on the pupil shape data at the current moment, the pupil position data is determined. The pupil position is the coordinates of the geometric center point calculated based on the pupil shape data. Using a pre-trained fixation point calculation model, the fixation point data at the current moment is obtained. Simultaneously, the Purkinje image can be used to assist in obtaining the fixation point data. It should be understood that the Purkinje image is a bright spot on the cornea of ​​the eye. It is produced by the reflection of light entering the pupil on the outer surface of the cornea.

[0051] The specific calculation process includes:

[0052] Based on multiple pupil calculation points (u, v), according to the ellipse feature formula The corresponding ellipse shape is calculated. An ellipse with a diameter between the minor axis (0.07) and major axis (0.29) of the inner and outer corners of the eye is selected as the pupil border. Quadrilaterals are formed using the endpoints of the major and minor axes, retaining only points whose average center lies inside the ellipse. The gradient of each point on the pupil outline edge is then calculated, and the ratio of the actual value of the gradient to the absolute value of the gradient is taken as the relative error of that point. After calculating the corresponding error value for each point on the outline, the overall average value is taken as an estimate of the effect for evaluation. During training, the Adam algorithm is used to iteratively update the parameters, using the mean squared error... As a loss function, the same calculation method as for the pupil is used to calculate the orbital contour data based on multiple orbital calculation points, which will not be elaborated here.

[0053] Furthermore, based on the current fixation point data and the initial fixation point data, the displacement of the current fixation point relative to the initial fixation point is calculated. The displacement of the current fixation point relative to the initial fixation point includes: the horizontal displacement and the vertical displacement of the current fixation point. Based on the current orbital contour data and the initial orbital contour data, the change in orbital area relative to the initial orbital contour is calculated (which can be represented by the ratio of orbital area). The horizontal displacement of the current fixation point is compared with a set horizontal threshold, the vertical displacement of the current fixation point is compared with a set vertical threshold, and the change in orbital area is compared with a set orbital area change threshold to obtain the comparison results for the current moment. It should be noted that when the horizontal displacement, vertical displacement, and change in orbital area are all expressed as percentages, the horizontal threshold, vertical threshold, and orbital threshold are generally set to 50%, but can be set to other values ​​according to actual needs; no limitation is made here.

[0054] Furthermore, to more accurately identify the user's camera module adjustment intentions and prevent damage to the camera device due to conflicting intentions, it is necessary to determine whether the current moment meets the stable adjustment conditions based on the comparison results at the current moment. Specifically, if the comparison result at the current moment is the same as the comparison results at the previous N moments, then the current moment is considered to meet the stable adjustment conditions. N is a positive integer. For example, if the horizontal displacement of the gaze point exceeds a horizontal threshold in the comparison result at the current moment, and the horizontal displacement of the gaze point also exceeds the horizontal threshold in the comparison results at the previous N moments, then the comparison result at the current moment is considered to meet the stable adjustment conditions.

[0055] Furthermore, given that the stable adjustment conditions are met at the current moment, based on the pre-acquired adjustment signal generation rules based on eye movements shown in Table 1 below, the camera module adjustment signal at the current moment is obtained according to the comparison results at the current moment.

[0056] Table 1. Rules for generating accommodation signals based on eye movements

[0057]

[0058] If the horizontal displacement of the gaze point exceeds a horizontal threshold in the current comparison result, a horizontal movement signal is generated to control the horizontal movement of the drone's camera module. If the vertical displacement of the gaze point exceeds a vertical threshold but the horizontal displacement does not exceed the horizontal threshold, a vertical movement signal is generated to control the vertical movement of the drone's camera module. If only the change in the orbital area exceeds a set orbital area change threshold in the current comparison result, a focus adjustment signal is generated to adjust the focus of the drone's camera module. If the horizontal displacement of the gaze point does not exceed the horizontal threshold, the vertical displacement of the gaze point does not exceed the vertical threshold, and the change in orbital area does not exceed the set orbital area change threshold, no adjustment signal is generated, and the drone's camera module remains stable. In the event of pupil or orbital area loss, no adjustment signal is generated, and the drone's camera module remains stable.

[0059] It should be noted that the camera module adjustment signals mainly include the camera module horizontal movement signal, the camera module vertical movement signal, and the camera module focus adjustment signal. These three signals are generated in a specific order: the camera module horizontal movement signal is generated first, followed by the camera module vertical movement signal, and finally the camera module focus adjustment signal.

[0060] For example, the acquisition frequency of eye images is 0.1s. N in the above example equals 4. The camera module's focus control determines the size change of the user's right eye socket. When the user's right eye socket area increases by 50% and remains stable for half a second, a short-focus camera module focus adjustment signal is sent until the right eye socket size recovers or is no longer detectable; when the user's right eye socket area decreases by 50% and remains stable for half a second, a long-focus camera module focus adjustment signal is sent until the right eye socket size recovers or is no longer detectable.

[0061] The training process of the fixation calculation module will be described below:

[0062] The same subject fixates on M different calibrated target points, where M is a positive integer, and corresponding eye images are captured. The pupil position of each eye image is determined. The pupil position of each eye image is input into a quadratic polynomial to obtain the estimated fixation point value for each eye image, and then compared with the actual coordinates of each target point. The coefficients of the quadratic polynomial are calculated using the least squares method to obtain the fixation point calculation model.

[0063] The quadratic polynomial is:

[0064] ;

[0065] Where (X,Y) are the coordinates of the fixation point, and (x,y) are the pupil positions.

[0066] In one embodiment, to achieve more precise adjustment of the drone's camera module, facial movements captured by a facial capture unit are used to assist in controlling the camera device. The facial capture unit captures a facial image of the user at the current moment. The current facial image is input into a pre-trained facial recognition model to obtain the facial movement recognition result at the current moment. Based on facial movement adjustment rules, the drone's camera module is adjusted according to the facial movement recognition result at the current moment.

[0067] The training process of a facial recognition model is as follows: Figure 6The process includes: acquiring a facial dataset containing actions of the forehead, ears, jaw, and cheeks; performing grayscale conversion and gamma correction on the dataset to obtain a preprocessed dataset; annotating the preprocessed dataset with facial actions; and inputting the annotated dataset into the MTCNN model for training to obtain a facial recognition model. The facial recognition model constructs a three-layer image pyramid, building windows at different pixel scales, correcting errors, reconstructing the original image, and outputting features to achieve alignment detection, keypoint localization, and reduce noise interference in the data against a relevant image test library. Subsequently, based on the training results, relevant facial region features are extracted, and a regression model is constructed to reconstruct and recognize facial expressions. It should be understood that gamma correction is used to adjust the mapping relationship between image brightness and pixel values. After model testing, a camera is installed in a position that can fully capture facial expressions, and the model's ability to correctly recognize facial actions is tested. If it cannot recognize facial actions, it needs to be retrained and adjusted until the requirements are met.

[0068] The facial movement adjustment rules include: setting the right side of the face as the positive direction and the left side of the face as the negative direction. Twitching of the cheek muscles controls the drone's camera module to move horizontally, twitching near the cheekbone controls the drone's camera module to move vertically, and changes in facial area are used to control the rotation angle of the drone's camera module.

[0069] In one embodiment, based on pre-acquired control signal generation rules based on hand gestures, a hand gesture recognition model is used to generate the UAV attitude control signal at the current moment based on the hand image at the current moment. This includes: inputting the hand image at the current moment into a pre-trained hand gesture recognition model to obtain the gesture recognition result at the current moment; and obtaining the UAV attitude control signal at the current moment based on the pre-acquired control signal generation rules based on the gesture recognition result at the current moment.

[0070] In one embodiment, the method for training a hand gesture recognition model includes: acquiring a hand gesture training set and annotating each image in the hand gesture training set with gestures; using the annotated hand gesture training set and with the assistance of a normalized attention module, training an attention-based gesture recognition network to obtain a hand gesture recognition model; wherein the attention-based gesture recognition network includes: a feature extraction layer, an attention module, two fully connected layers, and an output layer connected in sequence; the output layer is also connected to an object detection network.

[0071] The following section will explain the specific process of hand recognition:

[0072] The hand motion capture unit acquires an image of the user's hand at the current moment and inputs it into a pre-trained hand recognition model to obtain the gesture recognition result at the current moment. The gesture recognition result includes: the active part and the degree of activity.

[0073] Furthermore, to more accurately identify the user's drone attitude control intentions and prevent conflicts that could damage the drone, it is necessary to determine whether the current moment meets the stable control conditions based on the gesture recognition result. Specifically, if the gesture recognition result at the current moment is the same as the gesture recognition results at the previous L moments, then the current moment is considered to meet the stable adjustment conditions. L is a positive integer. For example, if the gesture recognition result at the current moment is a thumb movement, and the gesture recognition results at the previous L moments are all thumb movements, then the current moment is considered to meet the stable adjustment conditions.

[0074] Furthermore, given that the stability adjustment conditions are met at the current moment, the UAV attitude control signal at the current moment is obtained based on the pre-acquired control signal generation rules based on hand movements and the gesture recognition results at the current moment.

[0075] The control signal generation rules based on hand gestures include: if the gesture recognition result is thumb movement, a drone ascent speed control signal is generated to control the drone's ascent speed; if the gesture recognition result is thumb inactivity and index finger movement, a drone forward speed control signal is generated to control the drone's forward speed; if the gesture recognition result is only wrist movement, a drone rightward speed control signal is generated to control the drone's rightward speed; if the gesture recognition result is no movement, or if it cannot be recognized, the drone is controlled to hover.

[0076] In one specific embodiment, the control signal generation rules based on hand movements can be further refined, as detailed in Table 2 below.

[0077] Table 2. Control signal generation rules based on hand movements

[0078]

[0079] To address the issue of precise control, the same parameter is controlled by the actions of multiple parts in this embodiment. For example... Figure 7As shown, taking the control of upward movement by the degree of bending of the right thumb joint as an example, the movement is detected at both the base of the thumb and the joint. When moving forward, for every 10-degree increase in the bending angle at the base of the right thumb, the upward speed increases by 10% of the maximum speed; for every 10-degree increase in the bending angle of the thumb joint, the upward speed increases by 1% of the maximum speed. The change at the thumb joint is continuously adjusted to ensure accuracy; while the bending angle at the base of the thumb is adjusted in steps. Similarly, in the control process of the other two angles, for every 10-degree increase in the bending angle at the base of the index finger, the forward speed of the drone increases by 10% of the maximum speed; the bending angle of the first joint of the index finger changes continuously, increasing the forward speed by 1% of the maximum speed for every 10-degree increase. Corresponding to the wrist in the diagram, the clockwise rotation of the wrist represents the clockwise rotation of the drone.

[0080] The following will be combined with the appendix Figure 8 The training process of the hand motion recognition model is explained in 9 parts:

[0081] A hand motion training set is obtained, and gesture annotations are performed on each hand image in the training set. Gesture annotation includes: annotation of the active part and annotation of the degree of activity. For example, on a hand image, the annotation might indicate thumb activity, with the thumb root bending increasing by 10%. An attention-based gesture recognition network is constructed, comprising: a feature extraction layer, an attention module, two fully connected layers, and an output layer connected sequentially; the output layer is also connected to an object detection network. The feature extraction layer first extracts gesture features from the hand images, obtaining a gesture feature vector with the number of pixels occupied by the circumscribed cuboid of the hand as its dimension. Subsequently, the feature vector enters the attention module, where it is expressed as... Global average pooling compresses spatial features. In the formula, z is the output signal, u is the input signal, and h and w represent the two dimensions being pooled. The output of the attention module is then fed into the calculation formula. Two fully connected layers capture the dependency parameters within each channel, and then the weight coefficients are multiplied into the feature map of the corresponding channel to obtain a feature map with channel attention. In the formula, z represents the result of the previous layer. For activation function, For ReLU function, , These represent the weights of the two fully connected layers. The output of the last connected layer is output through the output layer.

[0082] The output layer inputs the results into the object detection network, which uses a multi-layered grid to predict icons using bounding boxes within the grid. The residuals are calculated through a multi-stage, multi-layered convolutional local network, and the degree of overlap between the predicted bounding boxes and the actual targets is used as the evaluation criterion for the performance.

[0083] During training, the output of the output layer is input into a normalized attention module, whose parameters are optimized based on the results of this module. This module includes a channel attention submodule and a spatial attention submodule, which determine the importance of channels and spaces, respectively. The channel attention submodule uses a batch-normalized scaling factor to measure the channel variance, thereby quantifying the channel importance and redistribution coefficients. The spatial attention submodule works similarly, measuring the batch-normalized scaling factor for the importance of spatial pixels.

[0084] It should be noted that this embodiment integrates algorithms such as attention judgment, autonomous learning, and object detection to improve recognition accuracy and speed based on autonomous learning. The accuracy is improved by 50% compared to previous developments, and the average recognition time is reduced by 25%. At the same time, security and stability are ensured by adding default values.

[0085] This invention combines facial motion recognition, hand motion recognition, eye motion recognition, and VR display to achieve virtual reality display and multimodal haptic interactive control. At the same time, it generates drone control commands by comprehensively judging the movements of multiple positions (eyes, face, and hands) to improve the accuracy of continuous adjustment.

[0086] In one specific embodiment, an additional agricultural payload for spraying pesticides is mounted on the drone. Additional sensors detect the spray volume on the little and ring fingers. The degree of bending of the two joints of the little finger controls the tens digit of the spray amplitude, the first joint controls the units digit, and the first joint of the ring finger controls the spray switch. When the corresponding position is reached, the sensor detects that the amplitude exceeds a threshold, generates a signal, transmits it to the control module, and encodes it into a control signal.

[0087] Figure 10 This is a flowchart illustrating the unmanned aerial vehicle (UAV) control method provided in an embodiment of this application. Figure 10 As shown, the drone control method based on multimodal haptic interaction and virtual reality includes:

[0088] Step S1001: Acquire the current eye image and hand image;

[0089] Step S1002: Based on the pre-acquired adjustment signal generation rules based on eye movements, using an eye recognition algorithm, a camera module control signal is generated based on the current eye image to adjust the camera module of the UAV accordingly; and based on the pre-acquired control signal generation rules based on hand movements, using a hand movement recognition model, a UAV attitude control signal is generated based on the current hand image to control the attitude of the UAV accordingly.

[0090] It should be understood that the specific implementation process of each step has been described in detail in the above device embodiments, and will not be repeated here for the sake of brevity.

[0091] Similar to the above embodiments, the present invention also provides an unmanned aerial vehicle (UAV) system, the UAV system comprising: a UAV ground station control device based on multimodal motion interaction and virtual reality as described above, and a UAV connected to the UAV ground station control device based on multimodal motion interaction and virtual reality.

[0092] Figure 11 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 11 As shown, the electronic terminal includes at least one processor 1101, a memory 1102, at least one network interface 1103, and a user interface 1105. The various components in the device are coupled together via a bus system 1104. It is understood that the bus system 1104 is used to implement communication between these components. In addition to a data bus, the bus system 1104 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 11 The general will label all buses as bus systems.

[0093] The user interface 1105 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0094] It is understood that memory 1102 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0095] In this embodiment of the invention, the memory 1102 is used to store various types of data to support the operation of the electronic terminal 1100. Examples of this data include: any executable program for operation on the electronic terminal 1100, such as the operating system 11021 and application programs 11022; the operating system 11021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 11022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the UAV control method based on multimodal motion sensing interaction and virtual reality provided in this embodiment of the invention can be included in the application program 11022.

[0096] The methods disclosed in the above embodiments of the present invention can be applied to processor 1101, or implemented by processor 1101. Processor 1101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 1101 or by instructions in the form of software. The processor 1101 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1101 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 1101 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0097] In an exemplary embodiment, the electronic terminal 1100 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0098] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0099] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0104] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0105] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0107] In summary, this application provides a UAV ground station control device, method, and UAV system based on multimodal haptic interaction and virtual reality. The device includes a main control module, a multimodal interaction module for acquiring eye and hand images, and a VR display module. By introducing a multimodal haptic interaction fusion mechanism, the UAV ground station control device of this application achieves more precise and safer UAV control. Therefore, this application effectively overcomes various shortcomings of the prior art and has high industrial application value.

[0108] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A UAV ground station control device based on multimodal haptic interaction and virtual reality, characterized in that, Connecting to drones includes: Main control module, multimodal interaction module, and VR display module; The main control module is connected to the multimodal interaction module and the VR display module respectively; The multimodal interaction module is used to acquire eye and hand images at the current moment; The main control module is used to receive the video stream information of the current moment collected by the camera module of the UAV, and decode and re-encode the video stream information of the current moment to obtain the display data of the VR display module at the current moment; The main control module is also used to generate a camera module control signal based on the current eye image, using an eye recognition algorithm and a pre-acquired adjustment signal generation rule based on eye movement, so as to adjust the camera module of the UAV accordingly; and to generate a UAV attitude control signal based on the current hand image, using a hand movement recognition model and a pre-acquired control signal generation rule based on hand movement, so as to control the attitude of the UAV accordingly.

2. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 1, characterized in that, Based on pre-acquired rules for generating accommodation signals based on eye movements, and utilizing an eye recognition algorithm, the camera module control signals for the current moment are generated based on the current eye image, including: The eye image at the current moment is preprocessed to obtain the preprocessed eye image at the current moment; Using an eye recognition algorithm, the preprocessed eye image at the current moment is processed to obtain the gaze point data and orbital contour data at the current moment. The current fixation point data and orbital contour data are compared with the pre-acquired eye reference data to obtain the comparison results at the current moment. Based on the pre-acquired rules for generating adjustment signals based on eye movements, the camera module control signal for the current moment is obtained according to the comparison results at the current moment.

3. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 2, characterized in that, The eye recognition algorithm includes: Multiple pupil calculation points and multiple orbital calculation points are selected on the preprocessed eye image at the current moment. Based on the multiple pupil calculation points and the multiple orbital calculation points, the pupil shape data and the orbital contour data at the current moment are obtained. Based on the pupil shape data at the current moment, the fixation point data at the current moment is obtained using a pre-trained fixation point calculation model.

4. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 2, characterized in that, The rules for generating accommodation signals based on eye movements include: If the horizontal displacement of the gaze point exceeds the set horizontal threshold in the comparison result, a horizontal movement signal of the camera module is generated; If the horizontal displacement of the gaze point does not exceed the set horizontal threshold and the vertical displacement of the gaze point exceeds the set vertical threshold in the comparison results, then a vertical movement signal of the camera module is generated. If the only change in the orbital area in the comparison results exceeds the set threshold for orbital area change, then a focus adjustment signal for the camera module is generated.

5. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 1, characterized in that, Based on pre-acquired control signal generation rules based on hand gestures, and utilizing a hand gesture recognition model, the current UAV attitude control signals are generated based on the current hand image, including: The hand image at the current moment is input into a pre-trained hand action recognition model to obtain the gesture recognition result at the current moment; Based on the pre-acquired control signal generation rules based on hand gestures, the current drone attitude control signal is obtained according to the gesture recognition result at the current moment.

6. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 5, characterized in that, Methods for training hand motion recognition models include: Obtain a hand gesture training set and annotate each image in the hand gesture training set with gestures; Using the labeled hand action training set and with the assistance of a normalized attention module, a hand action recognition network based on the attention mechanism is trained to obtain a hand action recognition model. The gesture recognition network based on the attention mechanism includes: a feature extraction layer, an attention module, two fully connected layers, and an output layer connected in sequence; the output layer is also connected to an object detection network.

7. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 5, characterized in that, The control signal generation rules based on hand movements include: If the gesture recognition result is a thumb movement, then a drone ascent speed control signal is generated; If the gesture recognition result is that the thumb is inactive and the index finger is active, then a drone forward speed control signal is generated; If the gesture recognition result shows only wrist movement, a rightward speed control signal for the drone is generated.

8. The UAV ground station control device based on multimodal haptic interaction and virtual reality according to claim 1, characterized in that, The multimodal interaction module includes an eye-tracking unit for acquiring eye images and a hand motion capture unit for acquiring hand images; the multimodal interaction module also includes a face capture unit; wherein, the face capture unit is used to acquire a face image at the current moment and send it to the control module; the control module uses a face recognition model based on the face image at the current moment to obtain the face motion recognition result at the current moment, so as to assist in adjusting the camera module of the drone.

9. A method for controlling an unmanned aerial vehicle (UAV), characterized in that, Used to control drones, including: Capture current-time images of the eyes and hands; Based on pre-acquired adjustment signal generation rules based on eye movements, an eye recognition algorithm is used to generate a camera module control signal based on the current eye image, so as to adjust the camera module of the UAV accordingly; and based on pre-acquired control signal generation rules based on hand movements, a hand movement recognition model is used to generate a UAV attitude control signal based on the current hand image, so as to control the attitude of the UAV accordingly.

10. An unmanned aerial vehicle (UAV) system, characterized in that, The unmanned aerial vehicle (UAV) system includes: a UAV ground station control device based on multimodal motion interaction and virtual reality as described in any one of claims 1 to 8, and a UAV connected to the UAV ground station control device based on multimodal motion interaction and virtual reality.

Images