Flying backpack composite control system based on exoskeleton gestures

Abstract

The invention relates to a flight backpack composite control system based on exoskeleton gestures. The flight backpack composite control system comprises a gesture data training module, a gesture recognition module, a gesture instruction mapping module, a button instruction mapping module and a man-machine composite control module. The single control limitation of an existing flight knapsack is broken through, the exoskeleton structure advantage and the data acquisition capacity are fused and deeply coupled, high-precision, high-stability and high-real-time control of the flight knapsack is achieved, traditional control pain points are solved, the exoskeleton bearing assistance function is achieved, and physical output of an operator is reduced. Meanwhile, an exclusive gesture recognition model is constructed by means of an exoskeleton multi-dimensional sensor, so that the anti-interference capability is improved, the method is more adaptive to complex operation scenes, and the application boundary is widened.

Description

Technical Field

[0001] This invention relates to the field of flight backpack control technology, specifically a flight backpack composite control system based on exoskeleton gestures. Background Technology

[0002] With the continuous development of personal flight equipment and human-computer interaction technology, the demand for jetpacks in emergency rescue, special operations, and other fields is gradually increasing. People have put forward higher requirements for the safety, stability, and operational efficiency of jetpack control methods, which has promoted the application and development of new human-computer interaction technologies in the field of jetpack control.

[0003] Current flight backpack control methods mostly rely on traditional means such as joysticks, buttons, or attitude sensing. These methods are relatively simple, offer limited control freedom, and are susceptible to operator states and external environmental factors in complex flight environments, resulting in insufficient control precision and poor stability. Furthermore, some gesture recognition controls rely on vision or a single sensor, have limited anti-interference capabilities, and lack effective redundancy control and protection mechanisms, making it difficult to meet the control requirements of flight backpacks in practical applications.

[0004] Furthermore, current applications of exoskeletons in flight backpack control largely focus on the exoskeleton's inherent functionality, neglecting to leverage the gesture data it can collect to establish a gesture recognition model. Therefore, it is necessary to utilize the hardware advantages of exoskeletons and algorithm optimization to develop a composite control system for flight backpacks based on a dedicated exoskeleton design. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention aims to provide a composite control system for flight backpacks based on exoskeleton gestures, thereby improving the recognition accuracy, stability, and real-time performance during flight backpack control.

[0006] The present invention is achieved through the following technical solution.

[0007] This invention provides a composite control system for a flight backpack based on exoskeleton gestures, which includes a gesture data training module, a gesture recognition module, a gesture command mapping module, a button command mapping module, and a human-machine composite control module.

[0008] The gesture data training module indirectly collects the relative position information of the operator's fingers through a six-axis spatial position sensor set on the exoskeleton, and preprocesses the collected gesture data to form basic data for gesture training.

[0009] The gesture data training module includes:

[0010] The data acquisition unit is used to acquire the relative position information of the operator's fingers and hands;

[0011] The data training unit is used to combine a large number of gesture samples and to simulate and train the gesture data.

[0012] The model building unit is used to build a gesture recognition model based on the U-Net deep learning network.

[0013] The gesture recognition module is a gesture recognition model based on the U-Net deep learning network. It identifies and analyzes the operator's gestures and outputs the corresponding gesture category.

[0014] The gesture recognition module includes:

[0015] The real-time recognition unit is used to input the real-time gesture data collected by the exoskeleton into the gesture recognition model to achieve real-time recognition of the operator's current gesture.

[0016] The recognition output unit is used to output a category signal corresponding to the gesture.

[0017] The gesture command mapping module is used to map the gesture categories output by the gesture recognition module to preset flight backpack control commands, thereby generating control command signals corresponding to the gestures.

[0018] The gesture command mapping module includes:

[0019] The gesture command matching unit is used to match the corresponding flight backpack control commands based on the recognized gesture category;

[0020] The gesture command output unit is used to output flight backpack control command signals corresponding to gestures.

[0021] The button instruction mapping module is used to map the physical buttons set on the exoskeleton to preset flight backpack control instructions, thereby generating corresponding control instruction signals.

[0022] The button instruction mapping module includes:

[0023] A button instruction matching unit is used to match corresponding flight backpack control instructions to buttons;

[0024] The button command output unit is used to output the flight backpack control command signal corresponding to the button.

[0025] The human-machine composite control module is used to convert between gesture control commands and physical button control commands set on the exoskeleton, so as to achieve composite control of the two.

[0026] The human-machine composite control module includes:

[0027] Central control system: Used to process transmitted signals and enable the flight backpack to perform corresponding operations;

[0028] The button control unit is used to acquire the operator's button input information and transmit signals to the central control system to control the flight backpack;

[0029] The gesture control unit is used to convert gesture control commands into signals and transmit the signals to the central control system to control the flight backpack.

[0030] The mode switching unit, which consists of physical buttons on the exoskeleton, is used to coordinate and switch between gesture control commands and button control commands to improve the stability and reliability of the flight backpack control process.

[0031] Through the coordinated operation of the above modules, the present invention can achieve high-precision, stable and real-time control of the flight backpack, effectively overcoming the problems of insufficient gesture recognition accuracy and poor control stability in the prior art.

[0032] More specifically, a flight backpack composite control system based on exoskeleton gestures includes a gesture data training module, a gesture recognition module, a gesture command mapping module, a button command mapping module, and a human-machine composite control module.

[0033] The gesture data training module indirectly collects the relative position information of the operator's fingers through a six-axis spatial position sensor set on the exoskeleton, and preprocesses the collected gesture data to form basic data for gesture training.

[0034] The gesture recognition module, based on the model built by the gesture data training module, identifies and analyzes the operator's gestures and outputs the corresponding gesture category;

[0035] The gesture command mapping module maps the gesture categories output by the gesture recognition module with preset flight backpack control commands to generate control command signals corresponding to the gestures.

[0036] The button command mapping module maps the buttons on the exoskeleton to preset flight backpack control commands to generate corresponding control command signals.

[0037] The human-machine composite control module is used to convert between gesture control commands and physical button control commands set on the exoskeleton, so as to achieve composite control of the two.

[0038] Furthermore, the gesture data training module includes a data acquisition unit, a data training unit, and a model building unit;

[0039] The data acquisition unit is used to acquire the relative position information of the operator's fingers and hands;

[0040] The data training unit is used to simulate and train the gesture data by combining a large number of gesture samples.

[0041] The model building unit is used to establish a gesture recognition model based on the U-Net deep learning network. The U-Net deep learning network adopts an encoder-decoder symmetric architecture, which includes an encoder, a bottleneck layer and a decoder. The encoder extracts gesture features by downsampling, and the decoder recovers the feature dimension by upsampling. Furthermore, it fuses the shallow features of the encoder and the deep features of the decoder through skip connections.

[0042] Furthermore, the convolutional feature calculation formula of the U-Net deep learning network is as follows:

[0043] (1)

[0044] in For the input feature map size, The kernel size is [size]. For the number of fillers, Step size;

[0045] The encoder contains four downsampling units, each downsampling unit consisting of two 3×3 convolutions, batch normalization, and... Activation function composition, The activation function formula is:

[0046] (2)

[0047] Furthermore, after each downsampling unit, 2×2 max pooling is used to halve the feature map size and double the number of channels;

[0048] The decoder contains four upsampling units. Each upsampling unit achieves feature upsampling through 2×2 deconvolution. The deconvolution feature calculation formula is the same as the convolution feature calculation formula. Furthermore, skip connections are used to concatenate the encoder's corresponding layer features with the decoder's current layer features in the channel dimension. The number of channels after concatenation satisfies the following condition:

[0049] (3)

[0050] in The number of encoder output channels. The number of upsampled channels in the decoder is ultimately mapped to the gesture category dimension through a 1×1 convolution, outputting the predicted probability value.

[0051] Furthermore, when constructing the gesture recognition model, the model building unit adopts a combined loss function of Dice loss and binary cross-entropy loss, as shown in the formula:

[0052] (4)

[0053] in For the true label of the gesture category, To predict probabilities for the model, The number of feature dimensions. To smooth out terms and prevent the denominator from being zero;

[0054] The model's output layer uses the Sigmoid activation function, with the following formula:

[0055] (5)

[0056] The predicted values ​​are normalized to the [0,1] interval, and the probability values ​​of each gesture category are output.

[0057] Furthermore, the gesture recognition module includes a real-time recognition unit and a recognition output unit;

[0058] The real-time recognition unit is used to input the real-time gesture data collected by the exoskeleton's six-axis spatial position sensor into the gesture recognition model to achieve real-time recognition of the operator's current gesture.

[0059] The recognition output unit is used to calculate the gesture recognition confidence level, using the following formula:

[0060] (6)

[0061] in For the total number of gesture categories, when At that time, the output signal corresponds to the category of the gesture.

[0062] Furthermore, the gesture command mapping module includes a gesture command matching unit and a gesture command output unit; the gesture command matching unit is used to match preset flight backpack control commands according to the recognized gesture category; the gesture command output unit is used to convert the matched control commands into signals that can be recognized by the flight backpack central control system and output flight backpack control command signals corresponding to the gestures.

[0063] Furthermore, the button instruction mapping module includes a button instruction matching unit and a button instruction output unit; the button instruction matching unit is used to match preset flight backpack control instructions according to the trigger signals of the exoskeleton physical buttons, and the instructions corresponding to emergency control and function buttons have the highest priority; the button instruction output unit is used to convert the matched control instructions into signals that can be recognized by the flight backpack central control system, and output flight backpack control instruction signals corresponding to the buttons.

[0064] Furthermore, the human-machine composite control module includes a central control system, a button control unit, a gesture control unit, and a mode switching unit;

[0065] The central control system is used to process the transmitted gesture control signals and button control signals, and drive the flight backpack to perform corresponding operations.

[0066] The button control unit is used to acquire the operator's button input information and transmit the signal to the central control system.

[0067] The gesture control unit is used to convert gesture control commands into signals and transmit them to the central control system.

[0068] The mode switching unit is a physical button set on the exoskeleton, used to coordinate and switch between gesture control commands and button control commands, realizing the conversion between three modes: gesture single control, button single control, and composite control.

[0069] Furthermore, the six-axis spatial position sensor is distributed and mounted on each joint of the exoskeleton fingers, integrating three-axis acceleration and three-axis angular velocity sensing units, and calculates the three-dimensional spatial relative position information of the hand through integral calculation;

[0070] The formula for calculating the three-axis spatial displacement of the six-axis spatial position sensor is as follows:

[0071] (8)

[0072] in for Axial acceleration, initial velocity and initial displacement are both taken as 0.

[0073] The formula for calculating the joint space attitude angle is:

[0074] (9)

[0075] in To bypass The angular velocity of the axis, with the initial attitude angle set to 0;

[0076] Adjacent joint sensing nodes , The formula for the spatial relative position vector is:

[0077] (10)

[0078] The formula for relative distance is:

[0079] (11).

[0080] Furthermore, the gesture data training involves preprocessing the data using a Min-Max normalization method, with the following formula:

[0081] (12)

[0082] in To standardize data, This is the raw position data of the sensor. , These are the extreme values ​​of the data;

[0083] The preprocessed dataset is divided into training, validation, and test sets in a 7:1:2 ratio. The training set is used for model parameter learning, the validation set for hyperparameter tuning, and the test set for model performance evaluation. Model training then begins, starting with hyperparameter configuration. Optimal parameters are determined through Bayesian optimization, and an initial learning rate is set. The training epoch threshold is 1000, and the batch size is 16. Then, optimizer training is performed, using the Adam optimizer to update parameters. The core formula is as follows:

[0084] (13)

[0085] in , , For the loss gradient, Finally, training is executed by inputting samples in batches, calculating the loss through forward propagation, and resetting the parameters through backpropagation. In each round, the performance is evaluated using the validation set and the loss is recorded, while the validation loss and the current learning rate are monitored simultaneously.

[0086] The next step is model optimization, starting with loss monitoring and statistically verifying the rounds in which the loss did not decrease. ,like Maintain the learning rate and continue training; if The learning rate is adjusted to half of its original value, and training continues. There is also an early stopping mechanism: if the training epochs reach 1000, training stops and the optimal weights are saved. Finally, the model output is tested on a test set using the trained network. If the performance does not meet the target, it returns to the preprocessing stage for adjustment until the performance meets the target, and then the final model is output for real-time recognition. The formulas used for core metric verification are as follows:

[0087] (14)

[0088] in True positive False positive The model validation set was a false negative. .

[0089] Compared with existing technologies, the advantages of this invention are: This invention breaks through the limitations of single-control systems in existing flight backpacks, integrating the advantages of exoskeleton structure and data acquisition capabilities, deeply coupling the two to achieve high-precision, high-stability, and high-real-time control of the flight backpack, solving the pain points of traditional control, while also leveraging the load-bearing assistance function of the exoskeleton to reduce the operator's physical exertion. Simultaneously, relying on the exoskeleton's multi-dimensional sensors to build a dedicated gesture recognition model enhances anti-interference capabilities, making it more adaptable to complex work scenarios and broadening its application boundaries. Attached Figure Description

[0090] Figure 1 A schematic diagram of the product compatible with the system provided in an embodiment of the present invention is shown.

[0091] Figure 2 The diagram illustrates the overall workflow of the system provided in this embodiment of the invention.

[0092] Figure 3 This diagram illustrates the locations of key components in a product compatible with the system, as provided in an embodiment of the present invention.

[0093] Figure 4 The overall system architecture diagram provided by an embodiment of the present invention is shown.

[0094] Figure 5 A detailed diagram of the system gesture data training module provided in an embodiment of the present invention is shown.

[0095] Figure 6 A detailed diagram of the system gesture recognition module provided in an embodiment of the present invention is shown.

[0096] Figure 7 A detailed diagram of the system gesture command mapping module provided in an embodiment of the present invention is shown.

[0097] Figure 8 A detailed diagram of the system button instruction mapping module provided in an embodiment of the present invention is shown.

[0098] Figure 9 A detailed diagram of the human-machine composite control module of the system provided in an embodiment of the present invention is shown.

[0099] Figure 10 The flowchart of the model training process of the system gesture data training module provided in the embodiment of the present invention is shown.

[0100] The attached diagram is labeled as follows: 1. Central control center, 2. Mechanical exoskeleton hand, 3. Control center - right, 4. Control center - left, 5. Button control device, 6. Sensor - right, 7. Sensor - left. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0102] This invention provides a composite control system for a flight backpack based on exoskeleton gestures. The system uses a mechanical exoskeleton hand as a human-computer interaction carrier, integrates a six-axis spatial position sensor and control buttons, and combines a U-Net deep learning network to build a high-precision gesture recognition model to achieve composite control of gesture control and button control. This solves the problems of traditional flight backpacks having a single control method, low recognition accuracy, and weak anti-interference ability, and improves the real-time performance and stability of flight control. It can be widely used in complex flight scenarios such as emergency rescue and special operations.

[0103] Figure 1 A schematic diagram of the product compatible with the system provided in an embodiment of the present invention is shown.

[0104] The hardware carrier of this invention consists of a central control center 1 for a flight backpack and a mechanical exoskeleton hand 2. After the operator wears the adapted mechanical exoskeleton hand, the sensors and control buttons on the exoskeleton collect gesture information and button information, respectively. The information is transmitted to the central control center of the flight backpack in the form of electromagnetic waves. The central control center analyzes, processes, and executes the received signals, ultimately driving the flight backpack to complete the corresponding flight actions, realizing a closed loop of human-machine interactive flight control.

[0105] Figure 2 This diagram illustrates the overall workflow of the system provided in an embodiment of the present invention.

[0106] The overall workflow of the exoskeleton gesture-based flight backpack composite control system of this invention consists of four core steps, as follows:

[0107] S1: Gesture Recognition Model Construction Stage. The relative position information of the operator's fingers is collected using exoskeleton sensors to obtain raw gesture data. After preprocessing the raw data, the massive gesture sample data is trained and iteratively optimized based on the U-Net deep learning network. The model convergence is optimized using a loss function formula to construct a high-precision gesture recognition model, enabling accurate recognition and classification of various control gestures.

[0108] S2: Gesture Command Mapping Stage. A one-to-one mapping relationship is established between the various gesture categories identified by the model in S1 and the preset control commands of the flight backpack. A standardized gesture command mapping table is developed to achieve direct conversion of gesture actions into flight control commands.

[0109] S3: Button Command Mapping Stage. Control buttons are set on the button control device 5 of the exoskeleton. The trigger signals of the buttons are bound and mapped with the core control, emergency control and other commands of the flight backpack to form a standardized button command mapping table, which forms a composite command input system with gesture input;

[0110] S4: Composite Control Execution Phase. The operator inputs control commands to the system via gestures or physical buttons. The command signals are transmitted to the central control system of the flight backpack via the exoskeleton control center. The central control system judges and processes the command signals, driving the flight backpack to perform corresponding flight operations. At the same time, the operator can switch between three modes in real time: gesture control, button control, and composite control via a mode switching button.

[0111] The gesture recognition model is built on the U-Net deep learning network. This network adopts a symmetrical encoder-decoder architecture, combined with skip connections and feature fusion mechanisms, to adapt to the feature extraction and classification requirements of sensor spatial location data. The model training and recognition accuracy are optimized through loss functions, activation functions, and feature calculation formulas. The core principles are as follows:

[0112] Overall architecture and feature calculation: The U-Net network has a U-shaped structure, consisting of an encoder, a bottleneck layer, and a decoder. The encoder downsamples the preprocessed gesture position data and extracts shallow detail features to deep semantic features layer by layer. The convolution feature calculation formula is (1). The bottleneck layer strengthens the core distinguishing features of the gesture. The decoder increases the dimension through deconvolution and combines the shallow detail features of the encoder with the deep semantic features of the decoder through skip connections. Finally, the gesture category prediction probability is output to achieve accurate recognition of flight control gestures.

[0113] (1)

[0114] in For the input feature map size, The kernel size is [size]. For the number of fillers, Step size

[0115] Encoder and Activation Function: The encoder contains four downsampling units. Each unit consists of two 3×3 convolutions, batch normalization, and a ReLU activation function. The activation function formula is formula (2). After 2×2 max pooling downsampling, the feature map size is halved and the number of channels is doubled, completing the dimensionality reduction of the gesture features. The feature calculation formula of the pooling layer is consistent with the above convolution feature calculation formula (where... ).

[0116] (2)

[0117] Decoder and Feature Fusion: The decoder is symmetrical to the encoder, containing four levels of upsampling units. Feature upsampling is achieved through 2×2 deconvolution, and the size calculation formula for the deconvolution output feature map is the same as that for convolutional features. Skip connections are used to concatenate the features of the corresponding layer of the encoder with the features of the current layer of the decoder along the channel dimension. The number of channels after concatenation is:

[0118] (3)

[0119] in The number of encoder output channels. The decoder upsamples the number of channels, which are then mapped to the gesture category dimension via a 1×1 convolution, outputting the predicted probability value.

[0120] Loss function optimization convergence: A combined loss function of Dice loss and binary cross-entropy (BCE) loss is used to address the uneven distribution of gesture sample categories. The formula is as follows:

[0121] (4)

[0122] in For the true label of the gesture category, To predict probabilities for the model, The number of feature dimensions. To smooth out terms and prevent the denominator from being zero.

[0123] Output layer and probability normalization: The model output layer uses the Sigmoid activation function to normalize the predicted values ​​to the [0,1] interval, and obtains the probability values ​​of each gesture category. The formula is (5). The category with the highest probability value is selected as the recognition result. The recognition confidence formula is (6).

[0124] (5)

[0125] (6)

[0126] in For the total number of gesture categories, when When the value is ≥0.95, the corresponding gesture category signal is output to complete accurate recognition.

[0127] Batch normalization optimizes training: Batch normalization is introduced after the convolutional layer to reduce feature distribution shift and accelerate model convergence. The formula is as follows:

[0128] (7)

[0129] Among them Batch average, For batch variance, This is the scaling factor. The offset coefficient, To smooth out terms and prevent the denominator from being zero.

[0130] Figure 3 This diagram illustrates the locations of key components in a product compatible with the system, as provided in an embodiment of the present invention.

[0131] The mechanical exoskeleton hand 2 is divided into a left hand and a right hand, and the components of the left and right hands are symmetrically distributed. The right hand is equipped with a control center - right 3, sensors - right 6 (a total of 10) and a button control device 5, while the left hand is equipped with a control center - left 4, sensors - left 7 (a total of 10) and a button control device 5.

[0132] The control center incorporates a gesture data training module, a gesture recognition module, a gesture command mapping module, and a button command mapping module.

[0133] The button control device includes both function buttons for controlling the flight backpack and mode switching buttons, enabling flexible switching between button input and control modes.

[0134] The aforementioned sensor is a six-axis spatial position sensor, distributed across the joints of the exoskeleton's fingers. It integrates three-axis acceleration and three-axis angular velocity sensing units, working in conjunction with the control center to collect and calculate the operator's three-dimensional relative position information of the hand, obtaining a three-dimensional coordinate map to provide raw data for gesture recognition. By integrating and spatially transforming the collected acceleration and angular velocity physical quantities, it achieves precise quantification of joint position and posture. The core principle is as follows:

[0135] Three-axis spatial displacement calculation: By integrating the acquired linear acceleration twice, the displacement of the sensing node along the x / y / z axes is obtained, as shown in the formula:

[0136] (8)

[0137] in for Axial acceleration, initial velocity and initial displacement are both taken as 0.

[0138] Joint space attitude angle calculation: Integrate the acquired angular velocity once to obtain the angle around the joint. Axis attitude angle (roll angle) Pitch angle Yaw angle The formula is:

[0139] (9)

[0140] in To bypass The angular velocity of the axis, with the initial attitude angle set to 0.

[0141] 3D relative position calculation: Let the adjacent joint sensing nodes be... , Then the spatial relative position vector and relative distance formulas of the two nodes are respectively formula (10) and formula (11), and each sensing node is greater than The sampling frequency transmits the raw data to the control center to obtain a three-dimensional coordinate map. After time synchronization and coordinate calibration, the data is fused to generate a complete three-dimensional spatial position dataset of the hand, which serves as the basis for subsequent training or recognition.

[0142] (10)

[0143] (11)

[0144] Figure 4 The overall system architecture diagram provided by an embodiment of the present invention is shown.

[0145] Functionally, the control system is divided into a gesture data training module, a gesture recognition module, a gesture command mapping module, a button command mapping module, and a human-machine composite control module. The exoskeleton sensor module provides raw training data and real-time acquisition data for gesture recognition, while the buttons on the exoskeleton provide command input channels for button control. The gesture command mapping module and the button command mapping module respectively realize the conversion of gestures and buttons into control commands. The outputs of the two command mapping modules are connected to the human-machine composite control module, which completes the coordination, switching, and final execution of dual commands. Each module works independently and communicates with each other, forming a complete flight backpack composite control system.

[0146] Figure 5 A detailed diagram of the system gesture data training module provided in an embodiment of the present invention is shown.

[0147] The gesture data training module is the fundamental core of gesture recognition, comprising three sub-units: a data acquisition unit, a data training unit, and a model building unit. Its working logic is as follows:

[0148] The data acquisition unit receives the three-dimensional spatial relative position information collected by the exoskeleton sensors in real time, providing raw spatial position data for model training;

[0149] The data training unit combines massive samples to normalize, label, simulate, and iteratively train the original data;

[0150] The model building unit builds a gesture recognition model based on the training dataset and the U-Net deep learning network. The model training is completed by adjusting the network hyperparameters and optimizing the loss function. Finally, the high-precision gesture recognition model is output and sent to the gesture recognition module for real-time recognition.

[0151] Figure 6 A detailed diagram of the system gesture recognition module provided in an embodiment of the present invention is shown.

[0152] The gesture recognition module implements real-time gesture recognition based on the model output by the gesture data training module. It consists of two sub-units: a real-time recognition unit and a recognition output unit. Its working logic is as follows:

[0153] The real-time recognition unit records the operator's current gestures in real time. After standardizing the real-time gesture data collected by the exoskeleton sensor, it inputs it into the U-Net gesture recognition model to realize real-time feature extraction and recognition of the current control gesture.

[0154] The recognition output unit calculates the gesture recognition confidence score based on the model recognition results. When the confidence score reaches a preset threshold, it outputs the category signal and instruction information corresponding to the gesture and transmits them to the gesture instruction mapping module for the next instruction matching step.

[0155] Figure 7 A detailed diagram of the system gesture command mapping module provided in an embodiment of the present invention is shown.

[0156] The gesture command mapping module comprises two sub-units: a gesture command matching unit and a gesture command output unit. Its working logic is as follows:

[0157] The gesture command matching unit receives the gesture category signal output by the gesture recognition module, and accurately matches the recognized gesture category according to the preset standardized gesture command mapping table to determine the corresponding flight backpack control command.

[0158] The gesture command output unit converts the matched control commands into electrical signals that can be recognized by the central control system of the flight backpack, and outputs the corresponding flight control command signals.

[0159] Figure 8 A detailed diagram of the system button instruction mapping module provided in an embodiment of the present invention is shown.

[0160] The button instruction mapping module and the gesture instruction mapping module are set up in parallel. Each module contains two sub-units: a button instruction matching unit and a button instruction output unit. Its working logic is as follows:

[0161] The button command matching unit receives button trigger signals from the exoskeleton button control device and, based on a preset standardized button command mapping table, precisely matches the triggered buttons to determine their corresponding flight backpack control commands. Note that emergency buttons and function buttons have the highest priority.

[0162] The button command output unit converts the matched control commands into electrical signals that can be recognized by the central control system of the flight backpack, and outputs the corresponding flight control command signals.

[0163] Figure 9 A detailed diagram of the human-machine composite control module of the system provided in an embodiment of the present invention is shown.

[0164] The human-machine interface control module is the core execution layer of the entire control system and is key to realizing combined gesture and button control. It comprises four sub-units: a central control system, a button control unit, a gesture control unit, and a mode switching unit. Its working logic is as follows:

[0165] The button control unit receives the control command signal output by the button command mapping module, converts it into a transmission signal that matches the central control system, and transmits it to the central control system, providing a signal transmission channel for button control;

[0166] The gesture control unit receives the control command signal output by the gesture command mapping module, converts it into a transmission signal that matches the central control system, and transmits it to the central control system, providing a signal transmission channel for gesture control;

[0167] The mode switching unit is a physical button set on the button control device, which is used to coordinate and switch between gesture and button control commands to realize real-time switching between three control modes;

[0168] The central control system, as the core processing unit, receives transmission signals from the button and gesture control units, as well as mode switching instructions from the mode conversion unit. It processes various signals and ultimately generates standardized flight control commands to drive the flight backpack actuators to complete corresponding flight actions, thereby achieving precise control of the flight backpack.

[0169] Figure 10 The flowchart of the model training process of the system gesture data training module provided in the embodiment of the present invention is shown.

[0170] The model training strictly follows a closed-loop process of start-up, preprocessing, training, optimization, and output through dataset construction, hyperparameter tuning, and model optimization strategies to ensure the accuracy and robustness of the U-Net gesture recognition model, as detailed below:

[0171] Data preprocessing: First, standardization is performed, which uses Min-Max normalization to unify the data range, with the formula (12). Then, annotation and vector conversion are performed, which involves labeling the data with flight control command tags, determining the joint three-dimensional vectors through sensor spatial coordinates using the same formula (10), and unifying the model input data format as required.

[0172] (12)

[0173] in To standardize data, This is the raw position data of the sensor. , These are the extreme values ​​of the data;

[0174] Dataset partitioning: After preprocessing, the dataset is divided into training set (70%), validation set (10%), and test set (20%) according to the proportions. This provides a balanced dataset for subsequent training and validation stages, ensuring the objectivity of the evaluation.

[0175] Model training: First, hyperparameters are configured, and optimal parameters are determined through Bayesian optimization, with an initial learning rate. The training round threshold is 1000 and the batch size is 16. Then, the optimizer is trained, and the Adam optimizer is used to update the parameters. The core formula is (13). Finally, the training is executed. Samples are input in batches, forward propagation is used to calculate the loss, and backpropagation is used to update the parameters. In each round, the performance is evaluated and the loss is recorded using the validation set. The validation loss and the current learning rate are monitored synchronously.

[0176] (13)

[0177] in , , For the loss gradient,

[0178] Model optimization: The first step is loss monitoring, statistically verifying the number of rounds in which the loss did not decrease. ,like Maintain the learning rate and continue training; if The learning rate is adjusted to half of the original value and training continues; in addition, there is an early stop mechanism, which stops training and saves the optimal weights if the number of training rounds reaches 1000.

[0179] Model output: The trained network is tested on a test set. If the performance does not meet the target, the model is returned to the preprocessing stage for adjustment until the performance meets the target. The final model is then output for real-time recognition. The formulas used for validating the core metrics are as follows:

[0180] (14)

[0181] in True positive False positive, The model validation set was a false negative. .

[0182] Example

[0183] A composite control system for a flight backpack based on exoskeleton gestures includes a gesture data training module, a gesture recognition module, a gesture command mapping module, a button command mapping module, and a human-machine composite control module.

[0184] The gesture data training module indirectly collects the relative position information of the operator's fingers through a six-axis spatial position sensor set on the exoskeleton, and preprocesses the collected gesture data to form basic data for gesture training; the gesture data training module includes a data acquisition unit, a data training unit, and a model building unit.

[0185] The data acquisition unit is used to acquire the relative position information of the operator's fingers and hands;

[0186] The data training unit is used to simulate and train the gesture data by combining a large number of gesture samples.

[0187] The model building unit is used to establish a gesture recognition model based on the U-Net deep learning network. The U-Net deep learning network adopts an encoder-decoder symmetric architecture, which includes an encoder, a bottleneck layer and a decoder. The encoder extracts gesture features by downsampling, and the decoder recovers the feature dimension by upsampling. Furthermore, it fuses shallow features of the encoder and deep features of the decoder through skip connections.

[0188] The convolutional feature calculation formula of the U-Net deep learning network is as follows:

[0189] (1)

[0190] in For the input feature map size, The kernel size is [size]. For the number of fillers, Step size;

[0191] The encoder contains four downsampling units, each of which consists of two 3×3 convolutions, batch normalization, and... Activation function composition, The activation function formula is:

[0192] (2)

[0193] Furthermore, after each downsampling unit, 2×2 max pooling is used to halve the feature map size and double the number of channels;

[0194] The decoder contains four upsampling units. Each upsampling unit achieves feature upsampling through 2×2 deconvolution. The deconvolution feature calculation formula is the same as the convolution feature calculation formula. Furthermore, skip connections are used to concatenate the encoder's corresponding layer features with the decoder's current layer features in the channel dimension. The number of channels after concatenation satisfies the following condition:

[0195] (3)

[0196] in The number of encoder output channels. The number of upsampled channels for the decoder is ultimately mapped to the gesture category dimension through a 1×1 convolution, outputting the predicted probability value;

[0197] The gesture recognition module, based on the model built by the gesture data training module, identifies and analyzes the operator's gestures and outputs the corresponding gesture category; the gesture recognition module includes a real-time recognition unit and a recognition output unit.

[0198] The real-time recognition unit is used to input the real-time gesture data collected by the exoskeleton's six-axis spatial position sensor into the gesture recognition model to achieve real-time recognition of the operator's current gesture.

[0199] The recognition output unit is used to calculate the gesture recognition confidence level, using the following formula:

[0200] (6)

[0201] in For the total number of gesture categories, when At that time, output a category signal corresponding to the gesture;

[0202] The gesture command mapping module maps the gesture categories output by the gesture recognition module to preset flight backpack control commands, generating control command signals corresponding to the gestures. The gesture command mapping module includes a gesture command matching unit and a gesture command output unit. The gesture command matching unit matches preset flight backpack control commands based on the recognized gesture categories. The gesture command output unit converts the matched control commands into signals recognizable by the flight backpack central control system, outputting flight backpack control command signals corresponding to the gestures.

[0203] The button command mapping module maps buttons mounted on the exoskeleton to preset flight backpack control commands, generating corresponding control command signals. The module includes a button command matching unit and a button command output unit. The button command matching unit matches preset flight backpack control commands based on the trigger signals of the physical buttons on the exoskeleton, with emergency control and function buttons having the highest priority. The button command output unit converts the matched control commands into signals recognizable by the flight backpack's central control system, outputting flight backpack control command signals corresponding to the buttons.

[0204] The human-machine composite control module is used to convert between gesture control commands and physical button control commands set on the exoskeleton, so as to realize the composite control of the two; the human-machine composite control module includes a central control system, a button control unit, a gesture control unit, and a mode conversion unit;

[0205] The central control system is used to process the transmitted gesture control signals and button control signals, and drive the flight backpack to perform corresponding operations.

[0206] The button control unit is used to acquire the operator's button input information and transmit the signal to the central control system.

[0207] The gesture control unit is used to convert gesture control commands into signals and transmit them to the central control system.

[0208] The mode switching unit is a physical button set on the exoskeleton, used to coordinate and switch between gesture control commands and button control commands, realizing the conversion between three modes: gesture single control, button single control, and composite control.

[0209] In a further preferred embodiment, when the model building unit constructs the gesture recognition model, it adopts a combined loss function of Dice loss and binary cross-entropy loss, as shown in the formula:

[0210] (4)

[0211] in For the true label of the gesture category, To predict probabilities for the model, The number of feature dimensions. To smooth out terms and prevent the denominator from being zero;

[0212] The model's output layer uses the Sigmoid activation function, with the following formula:

[0213] (5)

[0214] The predicted values ​​are normalized to the [0,1] interval, and the probability values ​​of each gesture category are output.

[0215] In a further preferred embodiment, the six-axis spatial position sensor is distributed and mounted on each joint of the exoskeleton fingers, integrating a three-axis acceleration and a three-axis angular velocity sensing unit, and calculating the three-dimensional spatial relative position information of the hand through integral calculation;

[0216] The formula for calculating the three-axis spatial displacement of the six-axis spatial position sensor is as follows:

[0217] (8)

[0218] in for Axial acceleration, initial velocity and initial displacement are both taken as 0.

[0219] The formula for calculating the joint space attitude angle is:

[0220] (9)

[0221] in To bypass The angular velocity of the axis, with the initial attitude angle set to 0;

[0222] Adjacent joint sensing nodes , The formula for the spatial relative position vector is:

[0223] (10)

[0224] The formula for relative distance is:

[0225] (11).

[0226] In a further preferred embodiment, the gesture data training involves preprocessing the data using a Min-Max normalization method, with the following formula:

[0227] (12)

[0228] in To standardize data, This is the raw position data of the sensor. , These are the extreme values ​​of the data;

[0229] The preprocessed dataset is divided into training, validation, and test sets in a 7:1:2 ratio. The training set is used for model parameter learning, the validation set for hyperparameter tuning, and the test set for model performance evaluation. Model training then begins, starting with hyperparameter configuration. Optimal parameters are determined through Bayesian optimization, and an initial learning rate is set. The training epoch threshold is 1000, and the batch size is 16. Then, optimizer training is performed, using the Adam optimizer to update parameters. The core formula is as follows:

[0230] (13)

[0231] in , , For the loss gradient, Finally, training is executed by inputting samples in batches, calculating the loss through forward propagation, and resetting the parameters through backpropagation. In each round, the performance is evaluated using the validation set and the loss is recorded, while the validation loss and the current learning rate are monitored simultaneously.

[0232] The next step is model optimization, starting with loss monitoring and statistically verifying the rounds in which the loss did not decrease. ,like Maintain the learning rate and continue training; if The learning rate is adjusted to half of its original value, and training continues. Additionally, there's an early stopping mechanism: if the training epochs reach 1000, training stops and the optimal weights are saved. Finally, the model output is tested on a test set using the trained network. If the performance doesn't meet the criteria, it returns to the preprocessing stage for adjustment until the performance targets are met, at which point the final model is output for real-time recognition. The formulas used for validating core metrics are as follows:

[0233] (14)

[0234] in True positive False positive The model validation set was a false negative. .

[0235] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0236] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A composite control system for a flight backpack based on exoskeleton gestures, characterized in that, It includes a gesture data training module, a gesture recognition module, a gesture command mapping module, a button command mapping module, and a human-machine composite control module; The gesture data training module indirectly collects the relative position information of the operator's fingers through a six-axis spatial position sensor set on the exoskeleton, and preprocesses the collected gesture data to form basic data for gesture training. The gesture recognition module, based on the model built by the gesture data training module, identifies and analyzes the operator's gestures and outputs the corresponding gesture category; The gesture command mapping module maps the gesture categories output by the gesture recognition module with preset flight backpack control commands to generate control command signals corresponding to the gestures. The button command mapping module maps the buttons on the exoskeleton to preset flight backpack control commands to generate corresponding control command signals. The human-machine composite control module is used to convert between gesture control commands and physical button control commands set on the exoskeleton, so as to achieve composite control of the two.

2. The flight backpack composite control system based on exoskeleton gestures according to claim 1, characterized in that, The gesture data training module includes a data acquisition unit, a data training unit, and a model building unit; The data acquisition unit is used to acquire the relative position information of the operator's fingers and hands; The data training unit is used to simulate and train the gesture data by combining a large number of gesture samples. The model building unit is used to establish a gesture recognition model based on the U-Net deep learning network. The U-Net deep learning network adopts an encoder-decoder symmetric architecture, which includes an encoder, a bottleneck layer and a decoder. The encoder extracts gesture features by downsampling, and the decoder recovers the feature dimension by upsampling. Furthermore, it fuses the shallow features of the encoder and the deep features of the decoder through skip connections.

3. The flight backpack composite control system based on exoskeleton gestures according to claim 2, characterized in that, The convolutional feature calculation formula of the U-Net deep learning network is as follows: （1） in For the input feature map size, The kernel size is [size]. For the number of fillers, Step size; The encoder contains four downsampling units, each downsampling unit consisting of two 3×3 convolutions, batch normalization, and... Activation function composition, The activation function formula is: （2） Furthermore, after each downsampling unit, 2×2 max pooling is used to halve the feature map size and double the number of channels; The decoder contains four upsampling units. Each upsampling unit achieves feature upsampling through 2×2 deconvolution. The deconvolution feature calculation formula is the same as the convolution feature calculation formula. Furthermore, skip connections are used to concatenate the encoder's corresponding layer features with the decoder's current layer features in the channel dimension. The number of channels after concatenation satisfies the following condition: （3） in The number of encoder output channels. The number of upsampled channels in the decoder is ultimately mapped to the gesture category dimension through a 1×1 convolution, outputting the predicted probability value.

4. The flight backpack composite control system based on exoskeleton gestures according to claim 2, characterized in that, When constructing the gesture recognition model, the model building unit uses a combined loss function of Dice loss and binary cross-entropy loss, as shown in the formula: （4） in For the true label of the gesture category, To predict probabilities for the model, The number of feature dimensions. To smooth out terms and prevent the denominator from being zero; The model's output layer uses the Sigmoid activation function, with the following formula: （5） The predicted values ​​are normalized to the [0,1] interval, and the probability values ​​of each gesture category are output.

5. The flight backpack composite control system based on exoskeleton gestures according to claim 1, characterized in that, The gesture recognition module includes a real-time recognition unit and a recognition output unit; The real-time recognition unit is used to input the real-time gesture data collected by the exoskeleton's six-axis spatial position sensor into the gesture recognition model to achieve real-time recognition of the operator's current gesture. The recognition output unit is used to calculate the gesture recognition confidence level, using the following formula: （6） in For the total number of gesture categories, when At that time, the output signal corresponds to the category of the gesture.

6. The flight backpack composite control system based on exoskeleton gestures according to claim 1, characterized in that, The gesture command mapping module includes a gesture command matching unit and a gesture command output unit. The gesture command matching unit is used to match preset flight backpack control commands according to the recognized gesture category. The gesture command output unit is used to convert the matched control commands into signals that can be recognized by the flight backpack central control system and output flight backpack control command signals corresponding to the gestures.

7. The flight backpack composite control system based on exoskeleton gestures according to claim 1, characterized in that, The button instruction mapping module includes a button instruction matching unit and a button instruction output unit. The button instruction matching unit is used to match preset flight backpack control instructions according to the trigger signals of the exoskeleton physical buttons, and the instructions corresponding to emergency control and function buttons have the highest priority. The button instruction output unit is used to convert the matched control instructions into signals that can be recognized by the flight backpack central control system and output flight backpack control instruction signals corresponding to the buttons.

8. The flight backpack composite control system based on exoskeleton gestures according to claim 1, characterized in that, The human-machine composite control module includes a central control system, a button control unit, a gesture control unit, and a mode switching unit; The central control system is used to process the transmitted gesture control signals and button control signals, and drive the flight backpack to perform corresponding operations. The button control unit is used to acquire the operator's button input information and transmit the signal to the central control system. The gesture control unit is used to convert gesture control commands into signals and transmit them to the central control system. The mode switching unit is a physical button set on the exoskeleton, used to coordinate and switch between gesture control commands and button control commands, realizing the switching between three modes: gesture single control, button single control, and composite control.

9. The flight backpack composite control system based on exoskeleton gestures according to claim 1, characterized in that, The six-axis spatial position sensor is distributed and mounted on each joint of the exoskeleton fingers, integrating three-axis acceleration and three-axis angular velocity sensing units, and calculates the three-dimensional spatial relative position information of the hand through integral calculation; The formula for calculating the three-axis spatial displacement of the six-axis spatial position sensor is as follows: （8） in for Axial acceleration, initial velocity and initial displacement are both taken as 0. The formula for calculating the joint space attitude angle is: （9） in To bypass The angular velocity of the axis, with the initial attitude angle set to 0; Adjacent joint sensing nodes , The formula for the spatial relative position vector is: （10） The formula for relative distance is: （11）。 10. A composite control system for a flight backpack based on exoskeleton gestures according to claim 2, characterized in that, The gesture data training involves preprocessing the data using the Min-Max normalization method, with the following formula: （12） in To standardize data, This is the raw position data of the sensor. , These are the extreme values ​​of the data; The preprocessed dataset is divided into training, validation and test sets in a 7:1:2 ratio. The training set is used for model parameter learning, the validation set is used for hyperparameter tuning, and the test set is used for model performance evaluation. Next comes model training, starting with hyperparameter configuration. Optimal parameters are determined through Bayesian optimization, and an initial learning rate is set. The training epoch threshold is 1000, and the batch size is 16. Then, optimizer training is performed, using the Adam optimizer to update parameters. The core formula is as follows: （13） in , , For the loss gradient, ; Finally, training is executed. Samples are input in batches, forward propagation is used to calculate the loss, back propagation is used to update the parameters, and the performance is evaluated and the loss is recorded in each round using the validation set. The validation loss and the current learning rate are monitored simultaneously. The next step is model optimization, starting with loss monitoring and statistically verifying the rounds in which the loss did not decrease. ,like Maintain the learning rate and continue training; if The learning rate is adjusted to half of its original value, and training continues. There is also an early stopping mechanism: if the training epochs reach 1000, training stops and the optimal weights are saved. Finally, the model output is tested on a test set using the trained network. If the performance does not meet the target, it returns to the preprocessing stage for adjustment until the performance meets the target, and then the final model is output for real-time recognition. The formulas used for core metric verification are as follows: （14） in True positive False positive, The model validation set was a false negative. .

Images