A Civil Aircraft Cockpit Display System Based on Gesture Recognition Algorithm
By optimizing gesture recognition using the HSACO-NSGA-BP algorithm, the problems of low recognition accuracy and slow convergence speed of traditional BP neural networks in avionics systems are solved, achieving a highly efficient gesture recognition effect.
Patent Information
- Application Number
- CN202410347374.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-26
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-03-26
AI Technical Summary
Traditional BP neural network-based gesture recognition technology suffers from problems such as slow convergence speed, easy getting stuck in local minima, and missing or distorted gestures in avionics systems, resulting in low recognition accuracy.
The HSACO-NSGA-BP algorithm, combined with ant colony optimization and genetic algorithm, is used to optimize the weights of the neural network and improve the accuracy of gesture recognition by combining global and local search capabilities.
It significantly improved the accuracy of gesture recognition, optimized the convergence speed of the neural network and the population diversity of the genetic algorithm, and achieved a gesture recognition accuracy of 97.02%.
Smart Images

Figure CN118097789B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cockpit display system technology, and more specifically to a civil aircraft cockpit display system based on a gesture recognition algorithm. Background Technology
[0002] The ARINC 661 standard is a widely used standard in the field of avionics, aiming to promote innovation and interoperability in avionics display systems. As avionics continues to evolve, the ARINC 661 standard also needs to adapt to new technological trends. This includes next-generation neural network technologies, which excel in handling complex tasks and data analysis. In avionics systems, neural networks can be used to improve gesture recognition systems, enhancing their accuracy and performance. By using deep learning neural networks, pilot gestures can be interpreted and recognized more accurately, reducing the possibility of misoperation. This not only improves system usability but also enhances the pilot's interactive experience with the system.
[0003] Currently, gesture recognition technology based on neural network learning has become a highly anticipated research field. Neural networks, with their excellent self-learning and nonlinear adaptive capabilities, have been widely used to process the structural features, edge structures, and contours of gestures. However, traditional back propagation (BP) neural network models have some limitations, such as slow convergence speed and susceptibility to local minima. Even after preprocessing the gesture image, using only the BP neural network algorithm still easily leads to problems such as missing gestures, distortions, and low accuracy. Yusuf et al. proposed a method combining global optimization algorithms such as Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) to enhance the exploration and search capabilities of the solution space. Hao Wang et al. proposed the GA-BP neural network, which combines genetic algorithms and back propagation algorithms to optimize network weights and prevent getting trapped in local optima. Compared to traditional BP neural network algorithms, the GA-BP algorithm has stronger global search capabilities, helps avoid getting trapped in local minima, and is expected to alleviate problems such as gesture distortion and missing gestures. Among them, genetic algorithms employ multi-point parallel search, evolving generation by generation according to the principle of survival of the fittest to produce increasingly better approximate solutions, thus achieving numerous research results. As a derivative method of genetic algorithms, non-dominated sorting genetic algorithms (NSGA) are multi-objective optimization algorithms based on the Pareto optimality concept, proposed by Srinivas et al. in 1995. The NSGA-III algorithm, developed from NSGA and NSGA-II, can effectively handle complex problems regardless of the nature of the problem.
[0004] To optimize system convergence and global search performance, and improve the discriminative power of different gesture samples, this paper proposes a hybrid global-local ant colony optimization-non-dominated sorting genetic algorithm (HSACO-NSGA-BP), combining the advantages of ant colony optimization (ACO) and genetic algorithms. Experimental results show that the proposed HSACO-NSGA-BP achieves an average accuracy of 97.02% in gesture recognition tasks under the ARINC661 standard, outperforming existing benchmark algorithms, demonstrating the effectiveness of the algorithm. Furthermore, a cockpit display system for civil aircraft based on this gesture recognition algorithm is proposed. Summary of the Invention
[0005] This invention provides a cockpit display system for civil aircraft based on a gesture recognition algorithm. By recognizing gestures using the HSACO-NSGA-BP algorithm, the recognition accuracy is improved, thereby solving the problems in the background technology.
[0006] The specific technical solution of this invention is as follows: A cockpit display system for civil aircraft based on a gesture recognition algorithm, comprising: The display unit is built into the cockpit display screen; A gesture module is used to generate and recognize the driver's gestures, and to display the recognized commands through the display unit. The data module is built into the display unit. It establishes gestures through the gesture working module and then collects and processes the gesture data in the display unit.
[0007] As one technical solution of the present invention, the display unit displays or hides window components according to the received instructions, windowing the content to be displayed; therefore, when the user clicks the monitor in the display unit, the monitor is unaware of the screen content and cannot define the relevant window component. To obtain gesture information, the specific window needs to be identified first, then the rendered screen inside needs to be pointed to, its layer needs to be located, and the window components of the rendered layer need to be traversed to locate the specific related window component number. This method is similar to a computer desktop, where many pages are open, allowing for related operations when locating the desired page, while other pages remain inaccessible and do not interfere with each other.
[0008] As one technical solution of this invention, the gesture working module includes a touch trajectory fitting and vectorization module, a gesture recognition module, and a gesture library module. To reduce operation latency, the touch trajectory fitting and vectorization module, the touch gesture recognition module, and the gesture library module generally reside on the server side of a C / S architecture. Following the recommendations of SAE ARP60494, the server side should also add feedback effects that do not depend on the client side's operation behavior, especially for actions with explicit behaviors. This feedback has high timeliness and can improve the interactive experience to a certain extent. Since this discussion concerns the rationality of the interaction and functional evaluation, the main approach adopted is Model-Based Design (MBD).
[0009] As a technical solution of the present invention, the gesture library module includes 7 basic gestures, including quick swipe gesture, single click gesture, long press gesture, double click gesture, swipe gesture, pinch gesture and rotate gesture.
[0010] As one technical solution of the present invention, the data module includes two main parts: data acquisition and data processing.
[0011] As a technical solution of the present invention, the data acquisition in the data module usually includes environmental interference noise in the acquired raw signal, and considering that the finger has a certain area, a solution program is used to obtain an accurate single touch point; the solution process includes filtering and solution process, and finally identifies the target; In the case of multi-touch, the original signal always contains all the touch input of the gestures. The algorithm is used to decompose the mixed gesture data into multiple data streams.
[0012] As one technical solution of the present invention, the data processing in the data module is based on the HSACO-NSGA-BP algorithm, and the algorithm flow includes: S1: Parameter initialization. The BP neural network is determined based on the input and output of the dataset. The number of nodes in the hidden layers is calculated based on the number of nodes in the input and output layers. This will generate all elements consisting of the initial weights and thresholds. S2: Initialize the population, including population encoding and population generation. An initial candidate solution population is randomly generated based on binary encoding. The number of individuals in the population is N. The ACO algorithm optimizes the initial population, calculates the fitness value (objective function value) of each individual in the population, and evaluates the quality of each objective allocation scheme. S3: Tournament selection. The tournament selection method is used to select the best individuals in the population to become the first generation parents. This allows the best individuals to have a greater chance of being selected while retaining a certain degree of randomness to ensure diversity. S4: Crossover and mutation, adaptive crossover and mutation of the parent population to produce new offspring; S5: Population optimization after crossover and mutation, based on the ACO algorithm to expand the new offspring and obtain a new solution after secondary optimization; S6: Merge to obtain a new population. Merge the parent generation and the offspring generation to obtain a new population of size 2N. S7: Non-dominated ranking and individual selection, performing non-dominated ranking and reference point selection on new populations, including establishing non-dominated stratification of the population and population selection in non-dominated strata; S8: Determine the number of iterations. Check if T is greater than the maximum number of iterations. If not, then T = T + 1 and return to step S3; otherwise, the algorithm ends. S9: Input the optimization result into the BP neural network training, calculate the training error, and output the result when the error reaches the target requirement; otherwise, continue the steps to recalculate the error.
[0013] As one technical solution of the present invention, the tournament selection process is as follows: S3.1: Fitness is calculated for each individual participating in the tournament; the higher the fitness, the higher the score. S3.2: Randomly select two individuals from the population. In the binary tournament selection, each individual has an equal probability of being selected. S3.3: Compare the fitness values of the selected individuals, select the individual with the better fitness value as the winner, and select it into the next generation of the population; S3.4: Repeat S3.1 to S3.3 until the new population size reaches the size of the original population.
[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention processes the HSACO-NSGA-BP algorithm, including parameter initialization, cluster initialization, tournament selection, crossover, and mutation, to achieve a significant advantage in convergence speed compared to other algorithms. In 50 experiments with two different iteration counts, the average convergence time is 1.53s and 1.23s, respectively, representing speedups of 35.17% and 49.80% compared to the original NSGA-III algorithm. The time complexity is not significantly increased compared to the original NSGA-III algorithm, thus effectively improving the accuracy of gesture recognition and demonstrating a clear advantage in cockpit gesture recognition applications. Furthermore, it optimizes the slow convergence speed and susceptibility to local minima inherent in BP neural networks, as well as addressing the poor population diversity and uneven solution distribution issues in genetic algorithms. Attached Figure Description
[0015] Figure 1 This is a structural diagram of the cockpit display system based on ARINC661 of the present invention; Figure 2 This is a diagram of the display unit window display structure of the present invention; Figure 3 This is a schematic diagram of the gesture recognition workflow of the present invention; Figure 4 This is a schematic diagram of the gesture position point calculation and filtering of the present invention; Figure 5 This is a schematic diagram of the multi-touch data separation of the present invention; Figure 6 This is a flowchart of the HSACO-NSGA-BP algorithm of this invention; Figure 7 This is a comparison diagram of the HSACO-NSGA-BP algorithm of this invention with other algorithms. Detailed Implementation
[0016] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.
[0017] Example 1: like Figure 1-7 As shown, a cockpit display system for civil aircraft based on a gesture recognition algorithm includes: The display unit is built into the cockpit display screen; The gesture module is used to generate and recognize the driver's gestures, and displays the recognized commands through the display unit. The data module is built into the display unit. Gestures are established through the gesture module, and then the gesture data is collected and processed in the display unit.
[0018] The cockpit display system based on ARINC661 can be divided into two key parts: CDS and UA, such as Figure 1 As shown. The CDS (Content Controller System) is used to generate and manage cockpit displays, while also providing data interfaces for external input devices, receiving and processing control commands from the pilot. The UA (User Agent) is responsible for providing data sources and user interaction control commands to the CDS. Once the system is powered on, the CDS loads and parses the Definition Files (DFs), and then uses the UA to perform logical processing of the displays and update screen information in real time.
[0019] However, one of the key innovations of the ARINC 661 specification is the explicit separation of the image display and control functions of the cockpit display system. In this design, the primary responsibility of the display is to show or hide window components based on received instructions, without concerning itself with the specific content of these components. Therefore, when a user clicks or performs an operation, the display system cannot directly determine the window component related to the operation. The core concept in this architecture is "windowing," such as... Figure 2 As shown, the screen to be displayed is divided into windows, each with a specific function and purpose.
[0020] When a display unit shows or hides widgets according to received instructions, it windows the content to be displayed. Therefore, when a user clicks on the monitor within the display unit, the monitor is unaware of the screen content and cannot define the relevant widget. To obtain gesture information, the specific window needs to be identified first, then the user points to the rendered screen within it, locates its layer, and iterates through the widgets on the rendered layer to pinpoint the specific widget number. This method is similar to a computer desktop, where many pages are open. When locating the desired page, related operations can be performed, but other pages remain inaccessible and do not interfere with each other.
[0021] like Figure 3 As shown, the gesture processing module includes a touch trajectory fitting and vectorization module, a gesture recognition module, and a gesture library module. To reduce operation latency, the touch trajectory fitting and vectorization module, the touch gesture recognition module, and the gesture library module generally reside on the server side of a C / S architecture. Following the recommendations of SAE ARP60494, the server side should also add feedback effects that do not depend on the client-side operation behavior, especially for actions with explicit behaviors. This feedback is highly timely and can improve the interactive experience to a certain extent. Since this discussion focuses on the rationality of the interaction and functional evaluation, a model-based design (MBD) approach is primarily adopted.
[0022] like Figure 5 As shown, the data module consists of two main parts: data acquisition and data processing. The data acquisition in the data module usually includes environmental interference noise when acquiring raw signals. Considering that fingers have a certain area, a calculation program is used to obtain accurate single touch points. The calculation process includes filtering and calculation processes, and finally identifies the target. like Figure 6 As shown, in the case of multi-touch, the original signal always contains the touch input of all gestures. The algorithm is used to decompose the mixed gesture data into multiple data streams.
[0023] Gesture processing begins with the data module. The gesture library module includes seven basic gestures: quick swipe, single click, long press, double click, swipe, pinch, and rotate. These gesture areas support most gesture operations and are configured with corresponding event reporting types, as shown in Table 1. In reporting mode, reporting data requires first determining the gesture type, and then packaging the data according to the relevant gesture event structure, as shown in Table 2 below. Table 1 Gesture Event Structure Table
[0024] Table 2 Packing of Rotation Gesture Events
[0025] Taking the rotation gesture as an example, Table 2 presents a data package table, which is based on the two rotation gesture operation methods explicitly defined in the ARINC 661 specification. These two methods are as follows: (1) The user presses down with two fingers, one finger remaining stationary while the other finger rotates and slides around the stationary finger. In this scenario, when the user's two fingers press down and generate rotational displacement, the system generates an event A661_EVT_GESTURE_ROTATE with a state of A661_GESTURE_START. As the user's fingers move, the rotation angle changes, periodically generating gesture events with a state of A661_GESTURE_UPDATE. When the gesture operation is complete, the system generates a gesture event with a state of A661_GESTURE_END. In these events, RelPosX and RelPosY represent the reference point of the screen, i.e., the center of rotation, while the parameter RelAngle represents the rotation angle of the screen. A positive value indicates counterclockwise rotation.
[0026] (2) The user presses down two fingers, and then both fingers rotate and slide together around their center point in the same direction. In this method, the system will also generate an event A661_EVT_GESTURE_ROTATE when the user presses down two fingers and generates rotational displacement, with the state A661_GESTURE_START. Similarly, as the fingers move, the rotation angle will change, and a gesture event with the state A661_GESTURE_UPDATE will be generated periodically. Finally, when the gesture operation is completed, the system will generate a gesture event with the state A661_GESTURE_END. The meanings of RelPosX, RelPosY, and RelAngle are the same as in the above method.
[0027] like Figure 6 As shown, the data processing algorithm HSACO-NSGA-BP in the data module includes the following steps: S1: Parameter initialization. The BP neural network is determined based on the input and output of the dataset. The number of nodes in the hidden layers is calculated based on the number of nodes in the input and output layers. This will generate all elements consisting of the initial weights and thresholds. S2: Initialize the population, including population encoding and population generation. An initial candidate solution population is randomly generated based on binary encoding. The number of individuals in the population is N. The ACO algorithm optimizes the initial population, calculates the fitness value (objective function value) of each individual in the population, and evaluates the quality of each objective allocation scheme. S3: Tournament selection. The tournament selection method is used to select the best individuals in the population to become the first generation parents. This allows the best individuals to have a greater chance of being selected while retaining a certain degree of randomness to ensure diversity. S4: Crossover and mutation, adaptive crossover and mutation of the parent population to produce new offspring; S5: Population optimization after crossover and mutation, based on the ACO algorithm to expand the new offspring and obtain a new solution after secondary optimization; S6: Merge to obtain a new population. Merge the parent generation and the offspring generation to obtain a new population of size 2N. S7: Non-dominated ranking and individual selection, performing non-dominated ranking and reference point selection on new populations, including establishing non-dominated stratification of the population and population selection in non-dominated strata; S8: Determine the number of iterations. Check if T is greater than the maximum number of iterations. If not, then T = T + 1 and return to step S3; otherwise, the algorithm ends. S9: Input the optimization result into the BP neural network training, calculate the training error, and output the result when the error reaches the target requirement; otherwise, continue the steps to recalculate the error.
[0028] The tournament selection process is as follows: S3.1: Fitness is calculated for each individual participating in the tournament; the higher the fitness, the higher the score. S3.2: Randomly select two individuals from the population. In the binary tournament selection, each individual has an equal probability of being selected. S3.3: Compare the fitness values of the selected individuals, select the individual with the best fitness value as the winner, and select it into the next generation of the population; S3.4: Repeat S3.1 to S3.3 until the new population size reaches the size of the original population; since each selection is independent, the same individual may be selected multiple times in different rounds, and some individuals may never be selected in the tournament. This allows superior individuals to have a greater chance of being selected while retaining a certain degree of randomness to ensure diversity.
[0029] Then, the HSACO-NSGA-BP algorithm used in this invention is compared with other algorithms, and the data shown in Table 3 is obtained.
[0030] Table 3 Comparison of HSACO-NSGA-BP algorithm with other algorithms
[0031] As shown in Table 3, the HSACO-NSGA-BP algorithm has a significant advantage in convergence speed. The average convergence time of 50 experiments under two different iteration numbers is 1.53s and 1.23s, which are 35.17% and 49.80% faster than the original NSGA-III algorithm, respectively. Compared with the original NSGA-III algorithm, the time complexity has not increased significantly. Therefore, it effectively improves the accuracy of gesture recognition and has a significant advantage in cockpit gesture recognition applications.
[0032] like Figure 7 As shown, the HSACO-NSGA-BP algorithm consistently outperforms the other four algorithms, further validating the advantages of the proposed algorithm. During the 70th training epoch, the gesture recognition accuracy approaches both the highest accuracy and the lowest training loss, remaining stable in the final epochs. Furthermore, the difference between the highest and lowest gesture recognition accuracy is nearly 10%, indicating significantly smaller prediction model errors. This demonstrates that the HSACO-NSGA-BP neural network model exhibits good optimization performance, improving prediction accuracy.
[0033] The embodiments of the present invention are given for the purposes of illustration and description, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features.
Claims
1. A cockpit display system for civil aircraft based on a gesture recognition algorithm, characterized in that, include: The display unit is built into the cockpit display screen; The gesture module is used to generate and recognize the driver's gestures, and to display the recognized commands through the display unit. The gesture module includes a touch trajectory fitting and vectorization module, a gesture recognition module, and a gesture library module. The data module is built into the display unit. It establishes gestures through the gesture module and then collects and processes the gesture data in the display unit. The data module includes two main parts: data acquisition and data processing. The data processing in the data module is based on the HSACO-NSGA-BP algorithm, and the algorithm flow includes: S1: Parameter initialization. The BP neural network is determined based on the input and output of the dataset. The number of nodes in the hidden layer is calculated based on the number of nodes in the input layer and the number of nodes in the output layer, which will generate all elements consisting of the initial weights and thresholds. S2: Initialize the population, including population encoding and population generation. An initial candidate solution population is randomly generated based on binary encoding. The number of individuals in the population is N. The ACO algorithm optimizes the initial population and calculates the fitness value of each individual in the population as the objective function value, which is used to evaluate the quality of each objective allocation scheme. S3: Tournament selection, which uses a tournament selection method to select outstanding individuals from the population to become the first generation of parents; S4: Crossover and mutation, adaptive crossover and mutation of the parent population to produce new offspring; S5: Population optimization after crossover and mutation, based on the ACO algorithm to expand the new offspring and obtain a new solution after secondary optimization; S6: Merge to obtain a new population. Merge the parent generation and the offspring generation to obtain a new population of size 2N. S7: Non-dominated ranking and individual selection, performing non-dominated ranking and reference point selection on new populations, including establishing non-dominated stratification of the population and population selection in non-dominated strata; S8: Determine the number of iterations. Check if T is greater than the maximum number of iterations. If not, then T = T + 1 and return to step S3; otherwise, the algorithm ends. S9: Input the optimization result into the BP neural network training, calculate the training error, and output the result when the error reaches the target requirement; otherwise, continue the steps to recalculate the error. The tournament selection process in Season 3 is as follows: S3.1: Fitness is calculated for each individual participating in the tournament; the higher the fitness, the higher the score. S3.2: Randomly select two individuals from the population. In the binary tournament selection, each individual has an equal probability of being selected. S3.3: Compare the fitness values of the selected individuals, select the individual with the better fitness value as the winner, and select it into the next generation of the population; S3.4: Repeat S3.1 to S3.3 until the new population size reaches the size of the original population.
2. The civil aircraft cockpit display system based on gesture recognition algorithm according to claim 1, characterized in that: The display unit displays or hides window components according to the received instructions, thus windowing the content to be displayed.
3. A civil aircraft cockpit display system based on a gesture recognition algorithm according to claim 1, characterized in that: The gesture library module includes 7 basic gestures, including quick swipe gesture, single click gesture, long press gesture, double click gesture, swipe gesture, pinch gesture, and rotate gesture.
4. A civil aircraft cockpit display system based on a gesture recognition algorithm according to claim 1, characterized in that: The data acquisition in the data module typically includes environmental interference noise when acquiring raw signals. Considering that fingers have a certain area, a calculation program is used to obtain accurate single touch points. In the case of multi-touch, the original signal always contains all the touch input of the gestures. The algorithm is used to decompose the mixed gesture data into multiple data streams.
Citation Information
Patent Citations
Optimization of safflower seed extraction process based on ant colony optimization method and back propagation neural network
JP2025173470A
