Unmanned aerial vehicle navigation control method based on natural gesture and gaze intention fusion

Abstract

The invention discloses an unmanned aerial vehicle navigation control method based on natural gesture and gaze intention fusion. The method comprises the following steps: obtaining a dynamic natural gesture and gaze of an operator; obtaining natural gesture intention probability distribution and gazing force intention probability distribution by utilizing the dynamic natural gesture intention understanding model and the gazing force intention understanding model; fusing the natural gesture intention probability and the gazing force intention probability by using an individual intention fusion model to obtain a control intention of the operator for the unmanned aerial vehicle; a motion control instruction is obtained through the control intention and the surrounding environment information, the unmanned aerial vehicle executes the motion control instruction, and navigation control over the unmanned aerial vehicle through dynamic natural gestures and watching force of an operator is achieved. According to the invention, the control intention of the operator is transmitted by using the dynamic natural gesture and the gazing force of the operator, the limitation on the skill level of the operator is effectively reduced, the timeliness of complex tasks is improved, and the unmanned aerial vehicle navigation control method which is more visual, natural and easy to master is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and pattern recognition, and in particular relates to a drone navigation control method based on the fusion of natural gestures and gaze intention. Background Art

[0002] At present, the control of drones mainly relies on remote control devices, which has certain requirements on the operator's skill level and limits the user experience of ordinary users. Especially when drones are required to complete complex tasks, remote control operations have a greater constraint on the timeliness of the task.

[0003] In addition, although the current control method based on static specific gestures attempts to solve some problems in remote control equipment control, the restrictiveness and complexity of gestures have led to insufficient control accuracy and limited control range. The process of identifying static specific gestures and mapping them to drone actions has high requirements for users and is easily affected by environmental factors and interference, which limits its practical application scenarios.

[0004] Dynamic natural gestures can express individual behavioral intentions and are a common way of communication in daily life; gaze usually refers to a person's focus on a specific object or area, which can also express the corresponding individual intention. The fusion of dynamic natural gesture intentions and gaze intentions can fully and accurately express the individual's behavioral intentions. The drone accurately understands the individual's control intentions of the drone by identifying and integrating the natural gesture intentions and gaze intentions. At the same time, it perceives the current surrounding environment, then integrates the perception results with the individual intentions, completes the corresponding intention instructions, and realizes the control of the drone by dynamic natural gestures and gaze. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a drone navigation control method based on the fusion of natural gestures and gaze intention.

[0006] Technical solution: The drone navigation control method based on the fusion of natural gestures and gaze intention described in the present invention comprises the following steps:

[0007] (1) The drone’s camera collects the operator’s image information in real time to obtain the operator’s dynamic and natural gestures and gaze;

[0008] (2) constructing a dynamic natural gesture intention understanding model and a gaze intention understanding model, and respectively training the dynamic natural gesture intention understanding model and the gaze intention understanding model based on the gesture intention understanding dataset and the gaze intention understanding dataset, and obtaining the natural gesture intention probability distribution and the gaze intention probability distribution using the dynamic natural gesture intention understanding model and the gaze intention understanding model according to the dynamic natural gesture and the gaze obtained in step (1);

[0009] (3) constructing an individual intention fusion model, and using the individual intention fusion model to fuse the natural gesture intention probability and the gaze intention probability to obtain the operator's control intention for the drone;

[0010] (4) Using the control intention obtained in step (3) and the surrounding environment information of the UAV to obtain motion control instructions, the UAV executes the motion control instructions to achieve navigation control of the UAV by the operator's dynamic natural gestures and gaze.

[0011] Furthermore, in step (1), the dynamic natural gesture of the operator is the posture information and trajectory information of the continuous gesture in each frame image of the operator, and the operator's gaze is the continuous eye gaze change and line of sight estimation information in each frame image of the operator.

[0012] Furthermore, in step (2), the gesture intention understanding dataset collects natural gesture images of the operator when realizing various intentions, annotates the posture information and trajectory information of the natural gestures, and assigns corresponding intention labels; the gaze intention understanding dataset collects gaze images of the operator when realizing various intentions, annotates the eye gaze changes and line of sight estimation information, and assigns corresponding intention labels.

[0013] Furthermore, in step (2), the dynamic natural gesture intention understanding model includes a Resnet50 network, a Swin-T network, a feature alignment fusion module, a first fully connected layer and a first Softmax layer, wherein the Resnet50 network and the Swin-T network are respectively used to extract the spatial posture features F of the natural gesture. p With the time trajectory feature F t The feature alignment fusion module is used to transform the spatial posture feature F p With the time trajectory feature F t Perform feature alignment and fusion to obtain the aligned fusion feature F h The first fully connected layer and the first Softmax layer are used to utilize the aligned fusion feature F h Get the probability distribution P of natural gesture intention h .

[0014] Furthermore, the feature alignment fusion module combines the spatial posture feature F p With the time trajectory feature F t Perform feature alignment and fusion:

[0015] F hi =F pi +F ti

[0016] Among them, F pi and F ti represents the spatial posture characteristics and temporal trajectory characteristics of the operator's natural gesture in the i-th frame, respectively, and F hi represents the aligned features in the i-th frame, and the aligned features F of multiple consecutive frames hi Perform feature splicing to obtain the aligned fusion feature F h .

[0017] Furthermore, in step (2), the gaze intention understanding model includes a PVT-Tiny network, a VIT / B-16 self-attention network, a spatiotemporal alignment fusion module, a second fully connected layer, and a second Softmax layer, wherein the PVT-Tiny network and the VIT / B-16 self-attention network are used to extract the gaze change feature F c and the line of sight estimation feature F e The spatiotemporal alignment fusion module is used to align the attention change feature F c and the line of sight estimation feature F e Perform time-space alignment fusion to obtain the time-space alignment fusion feature F g The second fully connected layer and the second Softmax layer are used to utilize the spatiotemporal alignment fusion feature F g Get the probability distribution of attention intention P g .

[0018] Furthermore, the spatiotemporal alignment fusion module uses the following formula to fusion the attention change feature F c and the line of sight estimation feature F e Perform spatiotemporal alignment fusion:

[0019] F gi =F ci +F ei

[0020] Among them, F pi and F ti represents the spatial posture characteristics and temporal trajectory characteristics of the manipulator's individual natural gesture in the i-th frame, respectively, and F gi represents the aligned features in the i-th frame, and the aligned features F of multiple consecutive frames gi Perform feature splicing to obtain the spatiotemporal alignment fusion feature F g .

[0021] Furthermore, in step (3), the individual intention fusion model fuses the natural gesture intention probability and the gaze intention probability by the following formula:

[0022]

[0023] Among them, P j represents the probability of the jth intention after fusion, α and β are learnable hyperparameters, P hj represents the probability that the natural gesture corresponds to the jth intention, P gj represents the probability that the attention corresponds to the jth intention, and n is the total number of intentions.

[0024] Furthermore, the operator's control intention of the drone in step 3 is expressed by the following formula:

[0025]

[0026] Among them, I represents the operator's control intention of the UAV, P j represents the probability of the jth intention after fusion, and T represents the predetermined threshold.

[0027] Furthermore, in step (4), the surrounding environment information of the drone is the position and size information of objects in the surrounding environment of the drone.

[0028] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The drone navigation control method based on the fusion of natural gestures and gaze intentions of the present invention utilizes the operator's dynamic natural gestures and gaze to convey the operator's control intentions, effectively reduces the restrictions on the operator's skill level, improves the timeliness of complex tasks, and provides a more intuitive, natural and easy-to-master drone navigation control method. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The flowchart of the UAV navigation control method based on the fusion of natural gestures and gaze intention is shown;

[0030] Figure 2 To provide a dynamic and natural gesture intention understanding model;

[0031] Figure 3 A model for gaze intention understanding;

[0032] Figure 4 Flowchart for the fusion of intentions for natural gestures and gaze. DETAILED DESCRIPTION

[0033] Example 1

[0034] See also Figure 1As shown, the drone navigation control method based on the fusion of natural gestures and gaze intention of the present invention comprises the following steps:

[0035] Step S1: Using the image information collected in real time by the camera carried by the drone itself, the detection model is used to obtain the posture information and trajectory information of the continuous gestures of the drone operator in each frame image, the continuous eye gaze changes and line of sight estimation information, and the position and size information of objects in the drone's surrounding environment.

[0036] Step S2: Build a dynamic natural gesture intention understanding model based on the Resnet50 network and the Swin-T network, such as Figure 2 As shown in the figure, Swin-T is the abbreviation of the Tiny version of Swin-Transformer. The dynamic natural gesture intention understanding model extracts the spatial posture features F of natural gestures respectively. p With the time trajectory feature F t Then the spatial posture feature F p With the time trajectory feature F t Perform feature alignment and fusion, and use F h The alignment is as follows:

[0037] F hi =F pi +F ti

[0038] Among them, F pi and F ti represents the spatial posture characteristics and temporal trajectory characteristics of the manipulator's individual natural gesture in the i-th frame, respectively, and F hi represents the aligned and fused features in the i-th frame, and the aligned and fused features of multiple consecutive frames F i Perform feature splicing to construct the above-mentioned total alignment fusion feature F h . The feature F h Input to the fully connected layer and the Softmax layer to obtain the probability distribution P of natural gesture intention h .

[0039] Step S3: Construct a gaze intention understanding model based on the PVT-Tiny network and the VIT / B-16 self-attention network, such as Figure 3 As shown in the figure, PVT in the PVT-Tiny network is the abbreviation of the Tiny version of the Pyramid Vision Transformer; VIT / B-16 is the abbreviation of the Vision Transformer Base version with 16 Transformer blocks. The attention intention understanding model extracts the attention change features F cand the line of sight estimation feature F e Then, F c With F e Perform spatiotemporal alignment fusion, and use F g The alignment is as follows:

[0040] F gi =F ci +F ei

[0041] Among them, F ci and F ei They represent the individual attention change characteristics and line of sight estimation characteristics of the operator in the i-th frame, respectively, and F gi represents the aligned and fused features in the i-th frame, and the aligned and fused features of multiple consecutive frames F gi Perform feature splicing to construct the above-mentioned total alignment fusion feature F g . The feature F g Input to the fully connected layer and the Softmax layer to obtain the probability distribution P of gaze intention g .

[0042] Step S4: Constructing gesture intention understanding and gaze intention understanding datasets refers to collecting natural gestures and gaze images of individuals in realizing certain behavioral intentions through instruments. The posture information and trajectory information of continuous natural gestures, as well as the changes in eye gaze and line of sight estimation information are marked, and corresponding behavioral intention labels are assigned.

[0043] Step S4: Construct an individual intention fusion module to fuse the individual dynamic natural gestures and gaze intentions, so as to accurately and efficiently obtain the control intention of the operator on the drone. The fusion diagram is shown in the figure. Figure 4 The strategy for integrating dynamic natural gestures and gaze intent is as follows:

[0044]

[0045] Among them, P j represents the probability of the jth intention after fusion, α and β are learnable hyperparameters, P hj represents the probability that the natural gesture corresponds to the jth intention, P gj Indicates the probability that the gaze corresponds to the jth intention. The values ​​of α and β are obtained by jointly training the self-built natural dynamic gesture and gaze dataset. At the same time, in order to avoid misjudgment of intent, the present invention performs a threshold preset operation. Only when the maximum probability of a certain intent in the fusion intent probability distribution is greater than the preset threshold, it is a valid intent.

[0046]

[0047] Among them, I represents the final individual control intention output, P j represents the fusion probability distribution, and T represents the predetermined threshold.

[0048] Step S5: Construct a fusion judgment decision module to comprehensively judge the individual control intention obtained in step S4 and the position and size information of the objects in the drone's surrounding environment obtained in step S1, so as to plan the drone's motion endpoint, flight path, flight speed and flight attitude. The drone executes the relevant command information of the drone's motion endpoint, flight path, flight speed and flight attitude, thereby completing the operator's individual dynamic natural gesture intention and gaze intention to control the drone's navigation.

Claims

1. A drone navigation control method based on the fusion of natural gestures and gaze intention, characterized by: The following steps are involved: (1) The drone’s camera collects the operator’s image information in real time to obtain the operator’s dynamic and natural gestures and gaze; (2) constructing a dynamic natural gesture intention understanding model and a gaze intention understanding model, and respectively training the dynamic natural gesture intention understanding model and the gaze intention understanding model based on the gesture intention understanding dataset and the gaze intention understanding dataset, and obtaining the natural gesture intention probability distribution and the gaze intention probability distribution using the dynamic natural gesture intention understanding model and the gaze intention understanding model according to the dynamic natural gesture and the gaze obtained in step (1); (3) constructing an individual intention fusion model, and using the individual intention fusion model to fuse the natural gesture intention probability and the gaze intention probability to obtain the operator's control intention for the drone; (4) The control intention obtained in step 3 and the surrounding environment information of the UAV are used to obtain motion control instructions, and the UAV executes the motion control instructions to realize the navigation control of the UAV by the operator's dynamic natural gestures and gaze.

2. The method for unmanned aerial vehicle navigation and control based on the fusion of natural gestures and gaze intention according to claim 1, characterized in that: In step (1), the dynamic natural gesture of the operator is the posture information and trajectory information of the continuous gesture in each frame image of the operator, and the operator's gaze is the continuous eye gaze change and line of sight estimation information in each frame image of the operator.

3. The method for unmanned aerial vehicle navigation and control based on the fusion of natural gestures and gaze intention according to claim 1, characterized in that: In step (2), the gesture intention understanding dataset collects natural gesture images of the operator when realizing various intentions, annotates the posture information and trajectory information of the natural gestures, and assigns corresponding intention labels; the gaze intention understanding dataset collects gaze images of the operator when realizing various intentions, annotates the eye gaze changes and line of sight estimation information, and assigns corresponding intention labels.

4. The method for unmanned aerial vehicle navigation control based on the fusion of natural gestures and gaze intention according to claim 1, characterized in that: In step (2), the dynamic natural gesture intention understanding model includes a Resnet50 network, a Swin-T network, a feature alignment fusion module, a first fully connected layer and a first Softmax layer. The Resnet50 network and the Swin-T network are used to extract the spatial posture features F of the natural gesture, respectively. p With the time trajectory feature F t The feature alignment fusion module is used to transform the spatial posture feature F p With the time trajectory feature F t Perform feature alignment and fusion to obtain the aligned fusion feature F h The first fully connected layer and the first Softmax layer are used to utilize the aligned fusion feature F h Get the probability distribution P of natural gesture intention h .

5. The method for unmanned aerial vehicle navigation control based on the fusion of natural gestures and gaze intention according to claim 4, characterized in that: The feature alignment fusion module combines the spatial posture feature F p With the time trajectory feature F t Perform feature alignment and fusion: F hi =F pi +F ti Among them, F pi and F ti represents the spatial posture characteristics and temporal trajectory characteristics of the operator's natural gesture in the i-th frame, respectively, and F hi represents the aligned features in the i-th frame, and the aligned features F of multiple consecutive frames hi Perform feature splicing to obtain the aligned fusion feature F h .

6. The method for unmanned aerial vehicle navigation control based on the fusion of natural gestures and gaze intention according to claim 1, characterized in that: In step (2), the gaze intention understanding model includes a PVT-Tiny network, a VIT / B-16 self-attention network, a spatiotemporal alignment fusion module, a second fully connected layer, and a second Softmax layer. The PVT-Tiny network and the VIT / B-16 self-attention network are used to extract the gaze change feature F c and the line of sight estimation feature F e The spatiotemporal alignment fusion module is used to align the attention change feature F c and the line of sight estimation feature F e Perform time-space alignment fusion to obtain the time-space alignment fusion feature F g The second fully connected layer and the second Softmax layer are used to utilize the spatiotemporal alignment fusion feature F g Get the probability distribution of attention intention P g .

7. The method for unmanned aerial vehicle navigation control based on the fusion of natural gestures and gaze intention according to claim 6, characterized in that: The spatiotemporal alignment fusion module uses the following formula to fusion the attention change feature F c and the line of sight estimation feature F e Perform spatiotemporal alignment fusion: F gi =F ci +F ei Among them, F pi and F ti represents the spatial posture characteristics and temporal trajectory characteristics of the manipulator's individual natural gesture in the i-th frame, respectively, and F gi represents the aligned features in the i-th frame, and the aligned features F of multiple consecutive frames gi Perform feature splicing to obtain the spatiotemporal alignment fusion feature F g .

8. The method for unmanned aerial vehicle navigation and control based on the fusion of natural gestures and gaze intention according to claim 1, characterized in that: In step (3), the individual intention fusion model fuses the natural gesture intention probability and the gaze intention probability through the following formula: Among them, P j represents the probability of the jth intention after fusion, α and β are learnable hyperparameters, P hj represents the probability that the natural gesture corresponds to the jth intention, P gj represents the probability that the attention corresponds to the jth intention, and n is the total number of intentions.

9. The method for unmanned aerial vehicle navigation control based on the fusion of natural gestures and gaze intention according to claim 8, characterized in that: In step (3), the operator's control intention of the drone is expressed as follows: Among them, I represents the operator's control intention of the UAV, P j represents the probability of the jth intention after fusion, and T represents the predetermined threshold.

10. The method for drone navigation control based on the fusion of natural gestures and gaze intention according to claim 1, characterized in that: In step (4), the surrounding environment information of the drone is the position and size information of objects in the surrounding environment of the drone.

Images