Method and system for building man-machine interaction test environment of aircraft cockpit

Abstract

The invention discloses an aircraft cockpit man-machine interaction test environment building method and system, and the method comprises the steps: building a virtual reality test environment in a simulation engine; wherein the virtual reality test environment comprises an external virtual environment and a cockpit internal environment; the method comprises the following steps: in a virtual reality test environment, carrying out custom layout on an internal environment of a cockpit, configuring a digital twinborn model according to hardware equipment information in the internal environment of the cockpit after layout, and generating a video perspective area according to the digital twinborn model; when a man-machine interaction test is carried out, a test demand is obtained, and a flight simulation model is configured according to the test demand; and establishing a flight simulation model, a simulation engine and a real object console, completing a man-machine interaction test of the cockpit in combination with gesture recognition and real object control, and outputting a test result. According to the method, virtuality and reality are combined, the simulation degree and immersion are high, the test environment can be quickly customized, and the cost is controllable.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and more specifically, to a method and system for building a human-computer interaction test environment for an aircraft cockpit. Background Technology

[0002] Human-computer interaction (HCI) testing refers to the process of testing and evaluating the interaction between humans and device systems. Its aim is to ensure that users can complete the interaction accurately and efficiently, and obtain a good user experience. In HCI testing, testers simulate the behavior of real users, performing a series of tasks and operations to evaluate the performance of the product's user interface design, interaction methods, and other aspects. Testers typically record user behavior and feedback, and provide improvement suggestions based on the test results to enhance the interaction design of the product or system.

[0003] Aircraft cockpits have complex interface layouts, with significant differences in the interaction logic and functionality of their various modules. Furthermore, erroneous operations during interaction can have fatal consequences. Therefore, conducting human-machine interaction (HMI) testing on aircraft cockpits is crucial for reducing pilot fatigue and stress during flight, improving flight efficiency, and minimizing the probability of errors. Existing methods for constructing HMI testing environments for aircraft cockpits can be broadly categorized into three types: First, building the testing environment using physical control panels or components. While this method is closest to reality, it incurs high testing costs and places high demands on the testing environment due to the inflexibility of some physical instruments and displays. Second, simulating the environment entirely in a virtual environment. This method saves costs but can only provide a rough verification of the cockpit's interaction logic and cannot assess the interaction of hardware devices during actual use. Third, although methods combining physical components and simulations exist, the current inability to properly integrate hardware and virtual environments results in poor immersion during testing and significant differences between the testing process and real operation, thus affecting the validity of HMI test results. Summary of the Invention

[0004] This application provides a method and system for building a human-computer interaction test environment for an aircraft cockpit, which overcomes at least one technical problem existing in the prior art and provides a highly realistic, cost-controllable and comprehensive method for building a human-computer interaction test environment for an aircraft cockpit.

[0005] The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a method for building a human-machine interaction testing environment for an aircraft cockpit, including:

[0007] A virtual reality testing environment is built in the simulation engine; wherein, the virtual reality testing environment includes an external virtual environment and a cockpit interior environment;

[0008] In the virtual reality testing environment, the interior environment of the cockpit is customized and laid out. Based on the hardware device information of the interior environment of the cockpit after the layout, a digital twin model is configured, and a video perspective area is generated based on the digital twin model.

[0009] When conducting human-computer interaction testing, the testing requirements are obtained, and the flight simulation model is configured according to the testing requirements.

[0010] Establish communication links between the flight simulation model, the simulation engine, and the physical control console, and combine gesture recognition and physical control to complete the cockpit human-machine interaction test, and output the test results.

[0011] In some embodiments of this application, the step of building a virtual reality test environment in the simulation engine specifically includes:

[0012] Acquire map data, and generate the external virtual environment in the simulation engine based on the map data; wherein, the map data includes terrain information and building information, the external virtual environment includes a terrain model and a building model, and the simulation engine generates the terrain model based on the terrain information and the building model based on the building information;

[0013] Import the aircraft model, use 3D software to customize the aircraft model, obtain the cockpit interior environment of the virtual reality test environment, and configure the interaction logic of the relevant hardware devices in the cockpit interior environment; wherein, the interaction logic includes gesture interaction logic and corresponding animation feedback for each interaction.

[0014] In some embodiments of this application, the step of customizing the layout of the cockpit interior environment in the virtual reality testing environment specifically includes:

[0015] The virtual reality testing environment is accessed using virtual reality equipment, and the leg and hand posture information of the test personnel is collected in real time through leg and hand sensors.

[0016] Based on the preset correspondence between leg posture information, hand posture information and virtual control commands, the corresponding initial virtual control command is determined; wherein, the leg posture information includes leg position information and opening and closing posture, the opening and closing posture being a separated state or a closed state; the hand posture information includes hand position information, grasping posture, five-finger extended posture, left and right rotation posture, and up and down flipping posture; the initial virtual control command includes control components and control actions.

[0017] When the tester's left or right hand posture information is a double-grip posture and the leg posture information is in a folded state, the corresponding component is determined as the control component of the initial virtual control command based on the leg position information and the hand position information, and the control action of the initial virtual control command is selected based on the double-grip posture, so as to select the control component according to the initial virtual control command.

[0018] Once the control component is selected, the current hand posture information of the tester is obtained in real time, and the corresponding virtual control command of the component is determined based on the hand posture information, so as to make corresponding layout adjustments to the control component according to the virtual control command of the component.

[0019] If the hand posture information is a five-finger extended posture, then the control action of the virtual control command of the component is to move, and the control component moves to the designated position with the tester's hand, thereby realizing the position layout adjustment of the control component;

[0020] If the hand posture information is a left rotation posture, then the control action of the virtual control command of the component is to shrink, and the control component adjusts its size and layout according to the left rotation angle of the tester's hand.

[0021] If the hand posture information is a right rotation posture, then the control action of the virtual control command of the component is to enlarge, and the control component will be enlarged and its layout adjusted according to the right rotation angle of the tester's hand.

[0022] If the hand posture information is an up-and-down flip posture, then the control action of the virtual control command for the component is release, and the selected control component is released.

[0023] In some embodiments of this application, configuring the digital twin model based on hardware device information in the laid-out cockpit interior environment specifically includes:

[0024] During the configuration process, the 3D model of the hardware device in the simulation engine is rendered with a light blue material. The hardware device in the simulation engine establishes a communication link with the corresponding hardware device in the physical control console to realize the configuration of the digital twin model of the hardware device.

[0025] In some embodiments of this application, generating a video perspective region based on the digital twin model specifically includes:

[0026] The rendering area of ​​the digital twin model is keyed, and the keyed area is expanded according to the hardware device in the physical control console. The hardware device is placed in the expanded keyed area, and the expanded keyed area changes synchronously with the hardware device and the digital twin model to realize the configuration of the video perspective area.

[0027] In some embodiments of this application, obtaining test requirements and configuring the flight simulation model according to the test requirements specifically includes:

[0028] Obtain the test requirements and set the environmental parameters and aircraft parameters for this flight in the flight simulation model according to the test requirements; wherein, the environmental parameters include the latitude and longitude information of the takeoff airport, weather conditions, and visibility, and the aircraft parameters include the test aircraft model information, startup status, and fault status.

[0029] In some embodiments of this application, the flight simulation model, the simulation engine, and the physical control console communicate with each other via the UDP protocol.

[0030] In some embodiments of this application, the 3D model of the hardware device in the simulation engine includes interactive 3D cockpit components and non-interactive 3D cockpit components. The step of combining gesture recognition with physical control to complete the cockpit human-machine interaction test specifically includes:

[0031] When the tester controls the interactive 3D cockpit components in the simulation engine using gesture recognition, the simulation engine generates and sends corresponding interactive commands to the flight simulation model. The flight simulation model modifies the flight attitude parameters according to the interactive commands and sends the modified flight attitude parameters back to the simulation engine, controlling the 3D model of the hardware device in the simulation engine to perform corresponding movements.

[0032] When the tester physically operates the physical control console, the physical control console sends corresponding control commands to the simulation engine. The simulation engine controls the movement of the corresponding digital twin model according to the control commands, and at the same time, the physical control console sends corresponding control commands to the flight simulation model.

[0033] In some embodiments of this application, the test results include operation recordings, number of interactions, interaction duration, and fault reports.

[0034] Secondly, embodiments of this application provide a system for building a human-machine interaction testing environment for an aircraft cockpit, comprising:

[0035] The environment setup module is used to build a virtual reality test environment in the simulation engine; wherein, the virtual reality test environment includes an external virtual environment and a cockpit interior environment;

[0036] The environment configuration module is used to customize the layout of the cockpit interior environment in the virtual reality test environment, configure the digital twin model according to the hardware device information in the cockpit interior environment after layout, and generate a video perspective area according to the digital twin model.

[0037] The model configuration module is used to obtain test requirements during human-computer interaction testing and configure the flight simulation model according to the test requirements.

[0038] The testing module is used to achieve data communication between the flight simulation model, the simulation engine and the physical control console via the UDP protocol, and to complete the cockpit human-machine interaction test by combining gesture recognition and physical control, and output the test results.

[0039] Thirdly, embodiments of this application provide an apparatus for building a human-machine interaction test environment for an aircraft cockpit, comprising: a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it executes the method for building a human-machine interaction test environment for an aircraft cockpit as described in the first aspect.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that can be executed by a processor to complete the method for building a human-machine interaction test environment for an aircraft cockpit as described in the first aspect.

[0041] The beneficial effects of the embodiments of this application are as follows:

[0042] 1. Virtual-real integration: This application can set up specific hardware-in-the-loop (HIL) simulations according to test requirements, and at the same time provide a three-dimensional simulation device with real interactive logic in a virtual environment. Different virtual-real integrated test environments can be constructed for different test situations to ensure the comprehensiveness and feasibility of the test.

[0043] 2. Customizable: This application can quickly customize the virtual reality testing environment according to the user's needs. After the environment is built, the user can quickly modify the cockpit layout in the virtual reality space by dragging and dropping gestures, thereby meeting the human-computer interaction testing needs of different cockpit environments.

[0044] 3. Controllable cost: This application can adjust the number of hardware connected to the system according to the budget. Unconnected hardware will complete the complete test process through system simulation. In addition, this application does not require a large space to set up test equipment or to build a complete physical test environment, which greatly reduces the test cost.

[0045] 4. High immersion: This application uses digital twin and video perspective technology to accurately segment the physical control panel and visual area, so that the real-world operation screen and the virtual screen are integrated. For hardware devices not connected to the system, gesture recognition technology provides a realistic interaction method, so that users can maintain a high level of immersion throughout the testing process. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application;

[0048] Figure 2 A schematic diagram illustrating the construction of a virtual reality testing environment in a method for constructing a human-computer interaction testing environment for an aircraft cockpit, as provided in an embodiment of this application;

[0049] Figure 3 A schematic diagram illustrating the configuration of interaction logic in a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application.

[0050] Figure 4 A schematic diagram illustrating the custom layout adjustment of the cockpit interior environment in a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application.

[0051] Figure 5 A schematic diagram illustrating the configuration of a digital twin model in a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application.

[0052] Figure 6 This is a schematic diagram illustrating the generation of a video perspective area in a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application.

[0053] Figure 7 A schematic diagram illustrating the configuration of a flight simulation model in a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application.

[0054] Figure 8 A schematic diagram illustrating the data exchange between the flight simulation model, simulation engine, and physical control console in a method for building a human-computer interaction test environment for an aircraft cockpit provided in this application embodiment;

[0055] Figure 9 This is a schematic diagram illustrating the actual operation of building a virtual reality test environment in a method for building a human-computer interaction test environment for an aircraft cockpit, as provided in an embodiment of this application.

[0056] Figure 10 This is a schematic diagram of the components of a test environment construction system for human-computer interaction in an aircraft cockpit, provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0058] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0059] This application discloses a method for building a human-computer interaction test environment for an aircraft cockpit. Primarily used in the field of human-computer interaction testing for aircraft cockpits, it combines gesture recognition, video perspective, and digital twin technologies to construct a virtual-real hybrid human-computer interaction test environment for an aircraft cockpit in virtual reality space. The test environment can be modified according to hardware conditions, providing a highly immersive and realistic human-computer interaction test environment. Detailed descriptions follow.

[0060] Figure 1 This paper illustrates a method for building a human-machine interaction test environment for an aircraft cockpit according to an embodiment of this application. Figure 1 As shown, the method for building the human-machine interaction test environment for the aircraft cockpit includes the following steps:

[0061] Step S110: Build a virtual reality test environment in the simulation engine.

[0062] Specifically, the virtual reality testing environment includes the external virtual environment and the cockpit interior environment.

[0063] In some embodiments, such as Figure 2 As shown, map data is acquired, and an external virtual environment is generated in the simulation engine based on the map data. The map data includes terrain and building information, while the external virtual environment includes a terrain model and building models. In the simulation engine, a terrain model is generated based on the terrain information, and a building model is generated based on the building information. Specifically, in the simulation engine, map data based on the real-world environment is acquired through a plugin and used to generate the external virtual environment for testing. Terrain information is used to generate the terrain model, and building information is used to generate a simplified building model. For the aircraft model in the test environment, the aircraft model is imported, and 3D software is used to customize its configuration to obtain the cockpit interior environment of the virtual reality test environment. The interaction logic of the relevant hardware devices in the cockpit interior environment is then configured. This interaction logic includes gesture interaction logic and corresponding animation feedback for each interaction. Specifically, users can choose to import existing 3D aircraft models, 3D aircraft models built using Blender and 3DMAX (3D Studio Max), or custom 3D aircraft models of existing models to obtain the cockpit interior environment. Furthermore, for the instruments, joysticks, and other equipment in the completed cockpit environment, gesture interaction actions and corresponding animation feedback were set up, referencing the interaction methods of relevant components in a real aircraft cockpit. After completing the construction of the virtual reality test environment and configuring the interaction logic, users can wear virtual reality devices to view the environment and interact with components such as knobs, buttons, and joysticks in the environment through gestures, such as... Figure 3 As shown.

[0064] Step S120: In the virtual reality test environment, customize the layout of the cockpit interior environment, configure the digital twin model according to the hardware device information in the cockpit interior environment after layout, and generate a video perspective area according to the digital twin model.

[0065] After the virtual reality testing environment is set up, users can wear virtual reality devices to enter the environment and quickly adjust the position, size, and other layout elements of the dashboard, joysticks, and other equipment inside the cockpit using gestures, thus customizing the layout of the virtual reality testing environment.

[0066] In some embodiments, a virtual reality device is used to enter a virtual reality testing environment. The leg and hand sensors of the virtual reality device collect the leg posture information and hand posture information of the tester (i.e., the user) in real time. Based on the preset correspondence between the leg posture information, hand posture information and virtual control commands, the corresponding initial virtual control command is determined. The leg posture information includes leg position information and opening and closing posture, which can be a separated state or a closed state. The hand posture information includes hand position information, grasping posture, five-finger extension posture, left and right rotation posture, and up and down flipping posture. The initial virtual control command includes control components and control actions.

[0067] When the tester's left or right hand is in a double-grip posture and their legs are in a folded position, the initial virtual control command's control component's location range is first determined based on the leg position information. Then, within this range, the control component's position is precisely determined based on the hand position information. The component at that position is selected as the initial virtual control command's control component. The control action of the initial virtual control command is then selected based on the double-grip posture. The selected control component is outlined to indicate to the user that it can be modified. Figure 4 As shown. It should be noted and understood that the "dual grip posture" in this application refers to the hand sensor capturing at least two gripping actions by the user; in addition, the "merged state" in this application refers to the distance between the user's legs not exceeding a preset leg distance, which can be set according to the actual situation.

[0068] Once a control component is selected, the system acquires the tester's current hand posture information in real time. Based on this hand posture information, it determines the corresponding virtual control command for the component and adjusts the layout of the control component accordingly. The specific layout adjustment process is as follows:

[0069] If the hand posture information indicates an outstretched hand position, the virtual control command for the component is to move it. The control component moves with the user's hand to the designated position, thus adjusting the component's layout. In other words, when a control component is selected, if the hand sensor detects that the user's raised hand is in an outstretched hand position, the selected control component moves with the user's hand until the user's hand movement changes to another action, such as changing from an outstretched hand position to a flipping hand position. At this point, the movement of the control component stops, and the selected control component is released.

[0070] If the hand posture information indicates a leftward rotation, the virtual control command for the component is to shrink, adjusting the size and layout of the control component according to the leftward rotation angle of the user's hand. Conversely, if the hand posture information indicates a rightward rotation, the virtual control command for the component is to enlarge, adjusting the size and layout of the control component according to the rightward rotation angle of the user's hand. In other words, when a control component is selected, if the hand sensor detects a rotational hand movement, the selected control component adjusts its size according to the user's hand rotation until the user's hand movement changes to another action, such as changing from a rotational to a flipping motion. At this point, the size adjustment of the control component stops, and the selected control component is released. Leftward rotation results in shrinkage, rightward rotation results in enlargement, and the size adjustment method for rotational posture is stepless, allowing adjustment of the size variation by controlling the rotation angle.

[0071] If the hand gesture is a flip-up / flip-down gesture, the virtual control command for the component will be a release, and the selected component will be released. Regardless of the user's previous hand gesture, as long as the user's hand gesture changes to a flip-up / flip-down gesture, the selected component will be released, and its outline will no longer be displayed. If there are no currently selected components, no action will be taken.

[0072] After configuring and adjusting the layout of the virtual reality testing environment, a communication link needs to be established between the hardware devices in the simulation engine and the corresponding hardware devices in the physical control console to achieve the configuration of digital twin models of the hardware devices. Once the settings are complete, when the tester controls the hardware devices, the 3D model of the devices within the simulation engine will change synchronously, realizing the configuration of the digital twin models of the hardware devices. During the configuration process, the 3D model of the hardware devices within the simulation engine can be rendered using a light blue material, such as... Figure 5 As shown.

[0073] Furthermore, the rendering area of ​​the digital twin model is keyed, and the keyed area is expanded based on the hardware devices in the physical control console. The hardware devices are then placed within the expanded keyed area, and the expanded keyed area changes synchronously with the hardware devices and the digital twin model to achieve video perspective area configuration. That is, after configuring the digital twin model of the hardware devices, the simulation engine is configured to key the rendering area of ​​the digital twin model, and the expansion of the keyed area is set according to the actual hardware used. In actual use, the hardware devices are placed within the keyed area. When the hardware devices and the digital twin model move, the keyed area also changes synchronously, thus achieving synchronization between the keyed area and the hardware device's movement. This, combined with digital twin technology, generates a video perspective area, enabling the physical device to be integrated with the virtual reality environment through the video perspective area, such as... Figure 6 As shown.

[0074] Step S130: When conducting human-computer interaction testing, obtain the testing requirements and configure the flight simulation model according to the testing requirements.

[0075] In some embodiments, actual test requirements are obtained, and environmental and aircraft parameters for the flight are set in the flight simulation model according to these requirements. Environmental parameters include, but are not limited to, the latitude and longitude of the takeoff airport, weather conditions, and visibility. Aircraft parameters include, but are not limited to, the test aircraft model information, startup status, and fault status. In specific implementation, several aircraft parameters can be preset in the flight simulation model. Preset aircraft parameters can be directly selected according to actual test requirements. However, if a completely new aircraft model is used for testing, new aircraft parameters must be input and constructed. Additionally, as... Figure 7 As shown, in addition to setting parameters in the flight simulation model, some parameters also need to be set in the virtual reality test environment according to actual test requirements, including but not limited to latitude and longitude, weather conditions, visibility, and aircraft model.

[0076] Step S140: Establish communication between the flight simulation model, the simulation engine, and the physical control console, and combine gesture recognition and physical control to complete the cockpit human-machine interaction test and output the test results.

[0077] In this application, the flight simulation model, simulation engine and physical control console communicate with each other via User Datagram Protocol (UDP) to achieve data interoperability.

[0078] In some embodiments, such as Figure 8As shown, the 3D model of the hardware devices in the simulation engine includes interactive 3D cockpit components and non-interactive 3D cockpit components. When the tester controls the interactive 3D cockpit components in the simulation engine through gesture recognition, the simulation engine generates and sends corresponding interactive commands to the flight simulation model. The flight simulation model modifies the flight attitude parameters according to the interactive commands and sends the modified flight attitude parameters back to the simulation engine, controlling the 3D model of the hardware devices in the simulation engine to perform corresponding movements. When the tester physically manipulates the physical control console, the physical control console sends corresponding control commands to the simulation engine. The simulation engine controls the movement of the corresponding digital twin model according to the control commands, and at the same time, the physical control console sends corresponding control commands to the flight simulation model. That is, before the test begins, the flight simulation model, simulation engine, and physical control console are pre-started and communication is established, thereby generating a virtual-real hybrid human-computer interaction test environment in virtual reality space by combining the physical control console and the interactive 3D cockpit components. When the tester controls the interactive 3D cockpit components in the simulation engine through gesture recognition, the simulation engine sends corresponding commands to the flight simulation model. When the flight simulation model receives a control command, it changes its flight attitude accordingly and sends the changed attitude back to the simulation engine, controlling the movement of the 3D cockpit model within the engine. When the physical control console is used for physical manipulation, it sends commands to the simulation engine to control the movement of the digital twin model within the virtual reality testing environment, while simultaneously sending corresponding control commands to the flight simulation model. During data communication, Python or similar languages ​​can be used as intermediaries to view the content of the transmitted information.

[0079] During human-computer interaction testing, the 3D cockpit model, combined with interactive 3D components and a physical control console, encompasses all functions involved in flight control. Testers can use gestures to control the simulation equipment in the virtual reality environment, as well as the physical control console. During testing, testers can verify the rationality of the overall cockpit design through a complete flight process; they can also focus on testing specific flight conditions to verify the cockpit's interactivity under special circumstances and the operation of specific physical control console components.

[0080] Furthermore, the test results in this application include operation recordings, number of interactions, interaction duration, and fault reports. After completing the test, the system will output the operation recordings of the test personnel in the cockpit during the test, to facilitate further evaluation of the system's human-computer interaction performance. The system will also output the number of interactions and interaction durations between the test personnel and specific components to determine the interactivity of those components. In addition, the system will also report fault situations during the test to determine the system's stability.

[0081] The above describes the steps of a method for building a human-machine interaction test environment for an aircraft cockpit provided in this embodiment. The following describes the process of building a human-machine interaction test environment for an aircraft cockpit using the American Piper PA-28 Archer III aircraft as an example.

[0082] First, a virtual reality testing environment was set up. Map data was imported into the simulation engine, and terrain and building models were generated based on the map data. Then, the .big format 3D model file of the PA-28 Archer III aircraft was imported. Figure 9 As shown.

[0083] After completing the virtual reality test environment setup, the interaction logic of interactive components such as knobs, switches, and joysticks in the aircraft cockpit is set up, including the gestures required to interact with different components and the animation feedback of the components after the interaction is completed.

[0084] Then, in a virtual reality testing environment, the positions and dimensions of the components inside the cockpit were adjusted using gestures until they met the testing requirements.

[0085] The joystick hardware is connected to the simulation engine, the digital twin model is synchronized, and a keyed area is generated based on the digital twin model. Then, the position of the joystick hardware is adjusted until it basically overlaps with the keyed area.

[0086] After configuring the hardware, enter the flight simulation model and set the starting point of the flight to a domestic airport. Search for and set the PA-28ArcherⅢ signal from the existing aircraft parameters, and keep the weather conditions and other parameters at their default settings. In the simulation engine, set the starting position and weather conditions to be consistent with the flight simulation model.

[0087] After configuring the flight simulation model, start the flight simulation model, simulation engine, and physical control console, and run the Python code responsible for receiving, compiling, and sending data. Use the data read in Python to determine whether the current three-way communication is smooth, thus completing the configuration of the test environment.

[0088] After completing the virtual reality test environment configuration, the test will proceed to the simulated flight phase, realizing the test process of taking off, circling the airport twice, and landing in a PA-28 Archer III aircraft from a domestic airport.

[0089] After the testing is completed, a test result report is obtained, which includes operation video recordings, the number of interactions between the tester and each component, the duration of each interaction, and the system's fault conditions during the testing process.

[0090] Corresponding to the above method embodiments, this application also provides a system for building a human-machine interaction test environment for an aircraft cockpit, used to execute the steps of the method for building a human-machine interaction test environment for an aircraft cockpit in the above embodiments. For example... Figure 10 As shown, the aircraft cockpit human-machine interaction test environment construction system 200 includes: environment construction module 210, environment configuration module 220, model configuration module 230 and test module 240.

[0091] Specifically, the environment setup module 210 is used to build a virtual reality test environment in the simulation engine. This virtual reality test environment includes an external virtual environment and a cockpit interior environment.

[0092] The environment configuration module 220 is used to customize the layout of the cockpit interior environment in a virtual reality testing environment, configure the digital twin model based on the hardware device information in the cockpit interior environment after layout, and generate a video perspective area based on the digital twin model.

[0093] The model configuration module 230 is used to obtain test requirements and configure the flight simulation model according to the test requirements during human-computer interaction testing.

[0094] Test module 240 is used to realize data communication between the flight simulation model, simulation engine and physical control console through UDP protocol, and to complete the cockpit human-machine interaction test by combining gesture recognition and physical control, and output the test results.

[0095] It should be noted that the aircraft cockpit human-computer interaction test environment construction system provided in this application is based on the same concept as the aircraft cockpit human-computer interaction test environment construction method embodiment of this application, and the technical effects it brings are the same as those of the aircraft cockpit human-computer interaction test environment construction method embodiment of the application. For details, please refer to the description in the aircraft cockpit human-computer interaction test environment construction method embodiment of this application, which will not be repeated here.

[0096] This application also provides an apparatus for building a human-machine interaction test environment for an aircraft cockpit, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described embodiments of the method for building a human-machine interaction test environment for an aircraft cockpit. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described embodiments of the system for building a human-machine interaction test environment for an aircraft cockpit.

[0097] In one specific embodiment, the computer program can be divided into one or more modules, which are stored in memory and executed by a processor to complete the embodiments of this application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the aircraft cockpit human-machine interaction test environment construction device. For example, the computer program can be divided into an environment construction module, an environment configuration module, a model configuration module, and a testing module, with the specific functions of each module as follows:

[0098] The environment setup module is used to build a virtual reality test environment within the simulation engine. This virtual reality test environment includes an external virtual environment and a cockpit interior environment.

[0099] The environment configuration module is used to customize the layout of the cockpit interior environment in a virtual reality testing environment. Based on the hardware device information in the layout of the cockpit interior environment, it configures the digital twin model and generates a video perspective area based on the digital twin model.

[0100] The model configuration module is used to obtain test requirements and configure the flight simulation model according to the test requirements during human-computer interaction testing.

[0101] The testing module is used to achieve data communication between the flight simulation model, simulation engine and physical control console via UDP protocol, and to complete the cockpit human-machine interaction test by combining gesture recognition and physical control, and output the test results.

[0102] The device for setting up the human-machine interaction test environment for an aircraft cockpit can be a computing device such as a desktop computer, laptop, handheld computer, or cloud management server. Those skilled in the art will understand that the device may include, but is not limited to, a processor and memory, and may also include more or fewer components, or combinations of certain components, or different components. For example, the device may also include input / output devices, network access devices, buses, etc.

[0103] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor, or it can be any conventional processor.

[0104] The memory can be an internal storage unit of the aircraft cockpit human-machine interaction test environment construction device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital Card (SD card), or Flash Card. Furthermore, the memory can include both internal and external storage units. This memory is used to store computer programs and other programs or data required by the aircraft cockpit human-machine interaction test environment construction device. It can also be used to temporarily store data that has been output or will be output.

[0105] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0106] Those skilled in the art will recognize that the modules and algorithm steps of the various embodiments described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0107] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that can be executed by a processor to complete the method for building a human-machine interaction test environment for an aircraft cockpit as described in the above embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), etc.

[0108] In summary, this application discloses a method and system for building a human-computer interaction test environment for an aircraft cockpit. It can set up specific hardware-in-the-loop simulations according to test requirements, and simultaneously provide 3D simulation equipment with realistic interaction logic in a virtual environment. Different virtual-real combined test environments can be constructed for different test situations, ensuring the comprehensiveness and feasibility of the tests. Furthermore, this application allows for rapid customization of the virtual reality test environment according to user needs. After the environment is built, users can quickly modify the cockpit layout in the virtual reality space through gestures such as dragging, thereby meeting the human-computer interaction test requirements for different cockpit environments. Simultaneously, this application allows for adjusting the number of hardware connected to the system according to budget. Hardware not connected will complete the entire test process through system simulation. Moreover, this application does not require a large space for test equipment or the construction of a complete physical test environment, significantly reducing test costs. In addition, this application uses digital twin and video perspective technology to accurately segment the physical control panel and visual area, allowing the real-world control screen and virtual screen to merge. For hardware devices not connected to the system, gesture recognition technology provides a realistic interaction method, ensuring that users maintain a high level of immersion throughout the test process.

[0109] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0110] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for building a human-computer interaction test environment for an aircraft cockpit, characterized in that, include: A virtual reality testing environment is built in the simulation engine; wherein, the virtual reality testing environment includes an external virtual environment and a cockpit interior environment; In the virtual reality testing environment, the interior environment of the cockpit is customized and laid out. Based on the hardware device information of the interior environment of the cockpit after the layout, a digital twin model is configured, and a video perspective area is generated based on the digital twin model. When conducting human-computer interaction testing, the testing requirements are obtained, and the flight simulation model is configured according to the testing requirements. Establish communication links between the flight simulation model, the simulation engine, and the physical control console, and combine gesture recognition and physical control to complete the cockpit human-machine interaction test, and output the test results.

2. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The process of building a virtual reality testing environment within the simulation engine specifically includes: Acquire map data, and generate the external virtual environment in the simulation engine based on the map data; wherein, the map data includes terrain information and building information, the external virtual environment includes a terrain model and a building model, and the simulation engine generates the terrain model based on the terrain information and the building model based on the building information; Import the aircraft model, use 3D software to customize the aircraft model, obtain the cockpit interior environment of the virtual reality test environment, and configure the interaction logic of the relevant hardware devices in the cockpit interior environment; wherein, the interaction logic includes gesture interaction logic and corresponding animation feedback for each interaction.

3. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The process of customizing the interior layout of the cockpit environment within the virtual reality testing environment specifically includes: The virtual reality testing environment is accessed using virtual reality equipment, and the leg and hand posture information of the test personnel is collected in real time through leg and hand sensors. Based on the preset correspondence between leg posture information, hand posture information and virtual control commands, the corresponding initial virtual control command is determined; wherein, the leg posture information includes leg position information and opening and closing posture, the opening and closing posture being a separated state or a closed state; the hand posture information includes hand position information, grasping posture, five-finger extended posture, left and right rotation posture, and up and down flipping posture; the initial virtual control command includes control components and control actions. When the tester's left or right hand posture information is a double-grip posture and the leg posture information is in a folded state, the corresponding component is determined as the control component of the initial virtual control command based on the leg position information and the hand position information, and the control action of the initial virtual control command is selected based on the double-grip posture, so as to select the control component according to the initial virtual control command. Once the control component is selected, the current hand posture information of the tester is obtained in real time, and the corresponding virtual control command of the component is determined based on the hand posture information, so as to make corresponding layout adjustments to the control component according to the virtual control command of the component. If the hand posture information is a five-finger extended posture, then the control action of the virtual control command of the component is to move, and the control component moves to the designated position with the tester's hand, thereby realizing the position layout adjustment of the control component; If the hand posture information is a left rotation posture, then the control action of the virtual control command of the component is to shrink, and the control component adjusts its size and layout according to the left rotation angle of the tester's hand. If the hand posture information is a right rotation posture, then the control action of the virtual control command of the component is to enlarge, and the control component will be enlarged and its layout adjusted according to the right rotation angle of the tester's hand. If the hand posture information is an up-and-down flip posture, then the control action of the virtual control command for the component is release, and the selected control component is released.

4. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The step of configuring a digital twin model based on the hardware device information in the cockpit interior environment after layout specifically includes: During the configuration process, the 3D model of the hardware device in the simulation engine is rendered with a light blue material. The hardware device in the simulation engine establishes a communication link with the corresponding hardware device in the physical control console to realize the configuration of the digital twin model of the hardware device.

5. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The step of generating a video perspective region based on the digital twin model specifically includes: The rendering area of ​​the digital twin model is keyed, and the keyed area is expanded according to the hardware device in the physical control console. The hardware device is placed in the expanded keyed area, and the expanded keyed area changes synchronously with the hardware device and the digital twin model to realize the configuration of the video perspective area.

6. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The process of obtaining test requirements and configuring a flight simulation model based on those requirements specifically includes: Obtain the test requirements and set the environmental parameters and aircraft parameters for this flight in the flight simulation model according to the test requirements; wherein, the environmental parameters include the latitude and longitude information of the takeoff airport, weather conditions, and visibility, and the aircraft parameters include the test aircraft model information, startup status, and fault status.

7. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The flight simulation model, the simulation engine, and the physical control console communicate with each other via the UDP protocol.

8. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The 3D model of the hardware devices in the simulation engine includes interactive 3D components of the cockpit and non-interactive 3D components of the cockpit. The cockpit human-computer interaction test, which combines gesture recognition and physical control, specifically includes: When the tester controls the interactive 3D cockpit components in the simulation engine using gesture recognition, the simulation engine generates and sends corresponding interactive commands to the flight simulation model. The flight simulation model modifies the flight attitude parameters according to the interactive commands and sends the modified flight attitude parameters back to the simulation engine, controlling the 3D model of the hardware device in the simulation engine to perform corresponding movements. When the tester physically operates the physical control console, the physical control console sends corresponding control commands to the simulation engine. The simulation engine controls the movement of the corresponding digital twin model according to the control commands, and at the same time, the physical control console sends corresponding control commands to the flight simulation model.

9. The method for building a human-machine interaction test environment for an aircraft cockpit according to claim 1, characterized in that, The test results include operation recordings, number of interactions, interaction duration, and fault reports.

10. A system for building a human-computer interaction testing environment for an aircraft cockpit, characterized in that, include: The environment setup module is used to build a virtual reality test environment in the simulation engine; wherein, the virtual reality test environment includes an external virtual environment and a cockpit interior environment; The environment configuration module is used to customize the layout of the cockpit interior environment in the virtual reality test environment, configure the digital twin model according to the hardware device information in the cockpit interior environment after layout, and generate a video perspective area according to the digital twin model. The model configuration module is used to obtain test requirements during human-computer interaction testing and configure the flight simulation model according to the test requirements. The testing module is used to achieve data communication between the flight simulation model, the simulation engine and the physical control console via the UDP protocol, and to complete the cockpit human-machine interaction test by combining gesture recognition and physical control, and output the test results.

Images