Multi-modal interactive vehicle-mounted helmet system design method

Abstract

The invention provides a multi-modal interaction vehicle-mounted helmet system design method, and belongs to the technical field of vehicle-mounted helmet system design, and the method comprises the following steps: S1, aiming at information processing processes of passengers under different situation tasks, considering cognitive load and resource distribution problems, and designing a vehicle-mounted helmet man-machine interaction design principle for balancing cognitive resources; s2, making a multi-modal design strategy of the vehicle-mounted helmet system based on context awareness based on the human-computer interaction design principle of the vehicle-mounted helmet for balancing the cognitive resources in the step S1; and S3, designing a context awareness man-machine interaction system based on the multi-modal design strategy of the vehicle-mounted helmet system based on context awareness in the step S2. By the adoption of the design method of the multi-mode interactive vehicle-mounted helmet system, the information processing capacity and combat effectiveness of military passengers in a complex battlefield environment can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle-mounted helmet system design, and in particular to a multi-modal interactive vehicle-mounted helmet system design method. Background Art

[0002] With the rapid development of information technology and the increasing complexity of the modern battlefield environment, the informationized combat capability of military equipment has become one of the key factors in determining the outcome of a war. As an important part of military equipment, the vehicle-mounted helmet system provides the crew with key battlefield information and assists them in making quick and accurate decisions. However, the traditional vehicle-mounted helmet system has a single interaction mode and often relies only on visual interaction, which brings many problems in actual combat.

[0003] First, single visual interaction can easily cause driving fatigue, and staring at the display screen for a long time will cause distraction, which in turn affects the decision-making judgment of the occupants. Secondly, in a complex and changing battlefield environment, the occupants need to process a large amount of information from various dimensions and channels, and the traditional interaction method cannot meet this demand, resulting in limited information processing capabilities and increased cognitive load. In addition, the existing vehicle-mounted helmet system differs from traditional smart wearable devices in application scenarios and interaction characteristics. Therefore, the demand for flexible configuration based on different complex situational tasks and multimodal interaction is becoming increasingly prominent.

[0004] In order to solve the above problems, academia and industry have begun to explore the application of multimodal interaction in vehicle-mounted helmet systems. Multimodal interaction refers to the integration of information from multiple sensory channels such as vision, hearing, and touch to achieve a more natural and efficient human-computer interaction method. This method can not only improve the efficiency of information processing, but also reduce the cognitive load of the crew and improve combat effectiveness.

[0005] The multimodal interaction design based on situational awareness further considers the information processing process and cognitive resource allocation of the crew in different situational tasks. By perceiving and analyzing the battlefield environment in real time, the system can adaptively adjust the interaction mode to provide the most accurate and efficient information feedback to the crew. This design not only improves the intelligence level of the vehicle-mounted helmet system, but also enhances the crew's perception of the battlefield situation and decision-making efficiency.

[0006] However, the current research on the multimodal interaction design of vehicle-mounted helmet systems based on contextual awareness is still in its infancy, and a systematic theoretical system and design method have not yet been formed. Therefore, the present invention proposes a multimodal interactive vehicle-mounted helmet system design method, which aims to solve the problems of the existing vehicle-mounted helmet system's single interaction mode and insufficient information processing capability, and provide military crews with a more efficient and intuitive way of receiving and processing information. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-modal interactive vehicle-mounted helmet system design method, which can improve the information processing ability and combat effectiveness of military crew members in complex battlefield environments.

[0008] To achieve the above object, the present invention provides a method for designing a multimodal interactive vehicle helmet system, comprising the following steps: Step S1: considering the information processing process of the occupants in different situational tasks, cognitive load and resource allocation issues are considered, and the human-computer interaction design principle of the vehicle helmet is designed to balance cognitive resources; Step S2, formulating a multimodal design strategy for a vehicle helmet system based on context awareness based on the vehicle helmet human-computer interaction design principle of balanced cognitive resources in step S1; Step S3: designing a context-aware human-computer interaction system based on the context-aware vehicle-mounted helmet system multimodal design strategy of step S2.

[0009] Preferably, in step S1, different situational tasks include marching phase tasks and combat phase tasks, and combat phase tasks include attack tasks and warning tasks.

[0010] Preferably, in step S1, the design principle of vehicle-mounted helmet human-computer interaction for balancing cognitive resources is: Excessive use of a single channel leads to an imbalanced cognitive load. In this case, other channels should be introduced to achieve resource redistribution. For example, during the interaction of the vehicle system, a certain sensory channel (such as the visual channel) is used too much, causing the channel to be overloaded and exceeding the normal range of its processing capacity, resulting in a decrease in the efficiency of information transmission and may even cause cognitive fatigue or information processing errors in the occupants. In other words, when the visual or other channels are used too much, the occupants' cognitive load may exceed their processing capacity, thereby affecting task performance and safety.

[0011] Excessive use of multiple channels leads to information overload. At this time, reduce the number of channels to achieve a balanced state of cognitive load.

[0012] When the system enables multiple channels (such as vision, hearing and touch) for information transmission at the same time and these channels are overloaded, cognitive overload will occur. Excessive use of multiple channels may make it impossible for occupants to effectively process information from all channels, resulting in information confusion, distraction or slow decision-making, thereby increasing the risk of errors.

[0013] Preferably, the channel includes a visual channel, an auditory channel, and a tactile channel.

[0014] Preferably, in step S2, the multimodal design strategy of the vehicle-mounted helmet system based on contextual perception includes a visual modal design strategy, an auditory modal design strategy, a tactile modal design strategy and a multimodal combination design strategy; Among them, the visual modality design strategy includes the structure layer, framework layer and presentation layer. The structure layer defines the information architecture based on the associated attributes of information, the display type of information, and the operation process of the task; the framework layer is modularly designed based on visual laws and perceptual processing; the presentation layer is designed based on visual unity, environmental adaptability and semantic rationality; Auditory modality design strategies include clarity and loudness control, noise reduction capabilities, directional audio delivery, context-aware audio cues, communications, and command and control capabilities; Tactile modality design strategies include vibration feedback, temperature perception, pressure perception, tactile interface, and personalized adaptation; The multimodal combination design strategies include the first combination strategy, the second combination strategy and the third combination strategy. The first combination strategy is a combination of the visual modal design strategy and the auditory modal design strategy, the second combination strategy is a combination of the visual modal design strategy and the tactile modal design strategy, and the third combination strategy is a combination of the auditory modal design strategy and the tactile modal design strategy.

[0015] Preferably, in step S3, the context-aware human-computer interaction system includes a task context setting module, a multimodal design strategy module, and a feedback and evaluation module; The mission scenario setting module is used to set marching missions, attack missions and warning missions; The multimodal design strategy module is used to set a first combination strategy, a second combination strategy and a third combination strategy; The feedback and evaluation module is used to monitor and evaluate the effect of the interactive system in real time according to the mission context and multimodal design strategy. By collecting data such as crew feedback, operation efficiency, reaction time, cognitive load, etc., the interactive strategy is dynamically adjusted to optimize the performance of the system and improve the success rate of mission execution.

[0016] Therefore, the present invention adopts the above-mentioned multi-modal interactive vehicle helmet system design method, and the beneficial technical effects are as follows: (1) Improve information processing efficiency: Multimodal interaction: By integrating multiple sensory modalities such as vision, hearing, and touch, multi-dimensional and efficient information transmission is achieved, improving the crew's information processing speed in complex battlefield environments.

[0017] Situational awareness: Adaptively adjust the interaction mode according to the needs of different situational tasks to ensure that passengers can quickly obtain key information and reduce cognitive load.

[0018] (2) Enhance combat effectiveness: Optimize decision-making process: Multimodal interaction design helps crew members understand the battlefield situation more accurately, thereby making faster and more reasonable decisions.

[0019] Shorten reaction time: Through efficient interaction, crew members can respond to battlefield changes more quickly and improve overall combat effectiveness.

[0020] (3) Improving user experience: Personalized interaction: Provide personalized interaction methods based on the information processing habits and preferences of different passengers to improve user experience and satisfaction.

[0021] Reduce driving fatigue: Through the flexible configuration of multi-modal interaction, driving fatigue caused by a single mode can be avoided and the crew's continuous combat capability can be improved.

[0022] (4) Promoting technological innovation: Integration of advanced technologies: This method integrates advanced technologies such as situational awareness and multimodal interaction, providing new ideas and methods for the future development of vehicle-mounted helmet systems.

[0023] Promote industrial upgrading: The research results are expected to promote the industrial upgrading of human-computer interaction design of military equipment and improve the overall technical level. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a model of information processing; Figure 2 Design principles for human-computer interaction to balance cognitive resources; Figure 3 Design strategies for visual interaction of vehicle-mounted helmet systems; Figure 4 It is a context-aware multimodal human-computer interaction system; Figure 5 is the average task completion time under different interaction modes; Figure 6 is the average α / β value under different interaction modes in the T1 task context; Figure 7 is the average α / β value under different interaction modes in the T2 task context; Figure 8 is the average SC (μS) under different interaction modes; Fig. 9 LF / HF in different interaction modes in the T1 task context; Fig.10 is the average LF / HF under different interaction modes in the T2 task context. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0026] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0027] Embodiment 1 The present invention provides a method for designing a multi-modal interactive vehicle-mounted helmet system, which mainly includes the following steps: S1. Based on the interaction characteristics and methods of the current vehicle-mounted helmet system in combat, summarize the problems existing in the human-computer interaction of the current vehicle-mounted helmet system.

[0028] The current interactive features and methods of vehicle-mounted helmet systems in combat are as follows: 1. Comprehensiveness and high integration: The vehicle-mounted helmet system is a comprehensive system that includes a helmet, a helmet display system, a helmet tracking system, and other possible auxiliary equipment, such as headphones, microphones, etc. This system is designed to provide military personnel with a highly integrated and fully functional battlefield information system.

[0029] 2. Visual information-driven: The helmet display system (HMDS) is the core part of the vehicle-mounted helmet system. It is responsible for displaying key battlefield information directly in front of the crew in the form of images or videos. Through HMDS, soldiers can obtain important information such as navigation data, battlefield intelligence, and the location of friendly and enemy forces in real time. Visual information is its main form of interaction.

[0030] 3. Head tracking and positioning: The vehicle-mounted helmet system usually has a head tracking function, which can track the occupant's head movement in real time, ensure that the information display is consistent with the occupant's perspective, and provide a more immersive interactive experience.

[0031] 4. Potential for multimodal interaction: Although the current interaction method of the vehicle-mounted helmet system is mainly visual, as a comprehensive platform, it provides soldiers with more comprehensive battlefield perception and communication capabilities, and has the potential to introduce multimodal interaction (such as hearing, touch, etc.) to improve interaction efficiency and user experience.

[0032] 5. Situational awareness capability: The vehicle-mounted helmet system needs to be able to perceive and analyze battlefield situations, and adjust information display and interaction methods according to situational changes to adapt to different combat needs.

[0033] 6. Real-time and high efficiency: In combat, the real-time and high efficiency of information are crucial. The vehicle-mounted helmet system needs to be able to quickly and accurately transmit key combat information to help crew members make wise decisions in complex battlefield environments.

[0034] 7. Coordination and communication: In addition to information display, the vehicle-mounted helmet system also has the ability to coordinate and communicate with other military equipment and systems, such as combining with long-range weapon systems such as drones and missiles to provide combatants with real-time and accurate battlefield information support.

[0035] The current problems with human-computer interaction in vehicle-mounted helmet systems are as follows: 1. The crew basically uses a single mode to handle multiple tasks, which will significantly reduce task performance; 2. Using multiple physical operators in the car to complete control will cause distraction and pose a certain threat to the completion of tasks in the car; 3. Uploading and downloading tasks after receiving instructions through the auditory channel is prone to delayed operation; 4. In a complex battlefield environment, combatants need to "look directly" at their surroundings (current head-down operations will threaten their lives); 5. The response speed of single mode output under extreme working conditions is easily affected; 6. If a single mode has a technical response delay and information display is out of sync without compensation from other modes, the combat rhythm will be disrupted and the mission cannot be carried out in an orderly manner.

[0036] S2. Considering the information processing process of the crew in different situational tasks, cognitive load and resource allocation issues are considered to obtain the internal mechanism of multimodal interactive feedback.

[0037] Information processing theory emphasizes a series of complex processing steps from perception to behavior in describing the human cognitive process. This theoretical framework is also applicable to human-computer interaction in vehicle-mounted helmet systems, and special attention should be paid to multimodal input and output.

[0038] During the information processing process, the crew of the tank armored vehicle equipped with the vehicle helmet system mainly collects battlefield information through various sensory organs. This information is received through visual, auditory, tactile and other sensors and transmitted to the brain for further processing. The following is a detailed explanation of the process: Information collection stage.

[0039] The crew uses various sensors of the vehicle's helmet system, such as the helmet display system (HMDS), auditory receivers (such as headphones) and tactile feedback devices (such as vibrators) to collect key information from the battlefield environment. This information includes: Visual information: Through HMDS, crew members can obtain important image or video information such as navigation data, battlefield intelligence, and the locations of friendly and enemy forces in real time.

[0040] Auditory information: The headset receives key sound information such as voice commands, alarm sounds, etc. from the vehicle system or other communication devices.

[0041] Tactile information: Through vibrators on the helmet or seat, the occupants can receive tactile feedback such as enemy attack warnings and system status changes.

[0042] Information perception and understanding stage.

[0043] The brain initially processes and analyzes the information received, identifying important features and patterns. This stage involves decoding, classifying and integrating information to form a holistic understanding of the battlefield environment.

[0044] Visual perception: The brain decodes the images displayed by the HMDS and identifies key targets, terrain features, navigation instructions, etc.

[0045] Auditory perception: The brain decodes the sound information it receives and understands the meaning and urgency of voice commands and alarm sounds.

[0046] Tactile perception: The brain decodes tactile feedback and recognizes the type of information represented by the vibration pattern (such as warning, confirmation, etc.).

[0047] Decision and reaction stage.

[0048] Based on the perceived and understood information, the crew quickly forms instructions and uses motor organs to accurately operate the onboard system or tank armored vehicle to complete the designated task.

[0049] Decision making: The brain develops appropriate driving strategies or combat plans based on the battlefield environment and mission requirements.

[0050] Reaction execution: The crew uses their hands, feet and other motor organs to operate the tank and armored vehicle's control devices, such as the steering wheel, accelerator, brake, etc., to execute decisions.

[0051] like Figure 1 As shown, the contents in each box in the figure represent the various stages and elements of information processing, and the arrows indicate the direction of information flow.

[0052] Cognitive load and resource allocation.

[0053] Throughout the information processing process, cognitive load and resource allocation play a vital role. As external stimuli increase in number and complexity, the demand for attention also increases. Occupants need to flexibly allocate attention resources to ensure efficiency when processing multiple tasks.

[0054] Attention management: Occupants need to selectively focus on important information in the environment and ignore irrelevant information to improve the efficiency of information processing.

[0055] Resource allocation: Crew members need to rationally allocate cognitive resources, such as attention and memory, to meet the demands of different tasks.

[0056] Therefore, the information processing task of the crew in the tank armored vehicle equipped with the vehicle helmet system is a complex and dynamic psychological process, involving multiple stages of perception, understanding, decision-making and reaction. By properly managing cognitive load and resource allocation, the crew can process battlefield information more effectively and improve combat effectiveness.

[0057] S3. Based on the inherent mechanism of multimodal interactive feedback in step S2, design human-computer interaction design principles for vehicle-mounted helmets that balance cognitive resources.

[0058] In order to ensure the cognitive efficiency and accuracy of the occupants when performing tasks, excessive use of a single channel should be avoided and the advantages of cross-channel information processing should be fully utilized. This paper proposes a human-computer interaction design principle that balances cognitive resources, such as Figure 2 As shown, excessive use of a single channel will lead to an imbalance in cognitive load, and other channels need to be mobilized to take on resource allocation. Excessive use of channels will lead to cognitive overload, and the number of channels needs to be reduced to balance channel information allocation resources to achieve a moderate effect.

[0059] S4. Based on the design principle of vehicle-mounted helmet human-computer interaction with balanced cognitive resources in step S3, formulate a multimodal design strategy for vehicle-mounted helmet system based on contextual awareness. Based on this strategy, design a contextually aware human-computer interaction system that can adaptively output according to the modal combination under different contextual tasks.

[0060] Visual modal design strategy: The visual modal design strategy is explained from three aspects: structure layer, framework layer and presentation layer, and the concept of cognitive load is introduced into interface design, such as Figure 3 As shown in the figure, the requirements for the structural layer are to define the information architecture based on the associated attributes of information (heading, direction, instruction, rotation speed, speed, fare, radar terrain diagram), display type and task operation process; the modular layout of the interface is proposed at the framework layer, and the visual laws of the human eye and the perceptual processing principles of the brain are also used as specifications; the presentation layer proposes the requirements of visual unity, environmental adaptability and semantic rationality from the perspective of interface symbols and colors.

[0061] Auditory modality design strategy: The auditory modal design strategy for military vehicle-mounted helmet systems needs to take into account multiple aspects to ensure that the crew can accurately receive and understand key information in a combat environment. The following are the design strategies: (1) Matching of sound type and task requirements.

[0062] The type of auditory feedback should be selected according to the needs of different tasks in order to effectively convey information: Warning sound: High-frequency, short warning sounds are suitable for emergencies, such as enemy threats, approaching obstacles, etc., and can quickly grab the attention of the occupants. For example, a short and sharp warning sound can remind the occupants to take immediate action.

[0063] Prompt sounds: Gentle and low-frequency prompt sounds can be used for routine tasks or mission status feedback, such as target lock, mission progress update, etc., to avoid disturbing the crew's attention, but be enough to remind them to perform relevant operations.

[0064] Voice prompts: In complex tasks, voice prompts can clearly convey task information, such as navigation instructions, task instructions, etc. Voice prompts should be concise, clear, and unambiguous to reduce the burden of understanding.

[0065] (2) Sound intensity and frequency adjustment.

[0066] The intensity and frequency of auditory feedback should be dynamically adjusted according to the urgency of the task, the ambient noise level, and the complexity of the task: Environmental noise adaptation: In high-noise environments (such as combat environments, vehicle engine noise, etc.), the sound intensity needs to be enhanced to ensure the effective transmission of auditory information. At the same time, the frequency adjustment should ensure that the sound does not interfere with the environmental noise, but can stand out clearly.

[0067] Mission urgency adjustment: For high-urgency tasks (such as enemy attack warnings and fire counterattack instructions), the intensity and frequency of the sound should be high to ensure that the crew can immediately detect and take appropriate actions. For low-priority or routine tasks, the sound intensity should be moderate to avoid excessive interference with attention.

[0068] Task complexity adjustment: In complex tasks, excessive auditory information input should be reduced to ensure that important instructions can be highlighted. The system can adjust the length and details of voice prompts according to the complexity of the task to avoid distracting occupants with overly long or complex voice information.

[0069] (3) Clarity and conciseness of voice prompts.

[0070] Voice prompts are a crucial part of mission execution. Their clarity and conciseness directly affect the crew's decision-making and response speed: Concise and clear: Voice prompts should avoid lengthy or complex instructions to ensure that the crew can understand the information in the shortest possible time. Each voice prompt should be clear and concise, directly conveying key information, such as "target locked", "alert status", etc.

[0071] Remove noise interference: In voice prompts, unnecessary noise or accents should be removed as much as possible to avoid misunderstanding or omission of information. The voice content should be consistent and standardized to ensure that all passengers can understand it accurately.

[0072] (4) Personalization and adaptability of voice prompts.

[0073] Auditory feedback should be personalized according to different occupant needs, mission situations and workloads to improve system adaptability and effectiveness: Personalized adjustment: Occupants can adjust the volume, frequency and type of sound according to their personal hearing preferences or needs to ensure the clarity and effectiveness of information transmission. In particular, adjustable voice frequency or voice type (such as male or female voice prompts) should be provided for different occupant backgrounds and experiences.

[0074] Task adaptation: The system should automatically adjust the type and intensity of sound prompts according to the real-time changes of the task. For example, in complex task switching, the system can help occupants distinguish the urgency and priority of the current task by adjusting the frequency, volume or content of auditory information.

[0075] (5) Spatial perception of sound.

[0076] In combat environments or complex missions, the spatial localization of sound is crucial, helping crew members identify the source of the signal and respond quickly: Stereo and Direction: Different types of warnings or mission information are delivered to different spatial areas through stereo or virtual spatial sound effects. The directionality of sound (such as sound from the left or right) helps the crew determine the mission location and threat source.

[0077] Sound localization and spatial adaptation: Sound localization design should be combined with the occupants’ head movement, line of sight and spatial distribution of tasks to ensure that sound can be accurately located in different situations and enhance the occupants’ spatial perception and reaction ability.

[0078] Haptic modality design strategy: The tactile modal design strategy of military helmet systems aims to provide additional sensory information to the occupants to enhance their ability to perceive and respond to the environment. The following are the tactile modal design strategies: (1) Selection of tactile feedback type.

[0079] The design of tactile modality must first determine the type of feedback, and choose according to the complexity and urgency of the task and user needs: Vibration feedback: Vibration feedback is a common and effective tactile signal that is suitable for various mission scenarios. Vibrations of different intensities and frequencies can convey different levels of mission information or warnings. For example, at the beginning of a mission, a slight vibration can be used as a prompt, while in an emergency, a stronger vibration is used to attract the driver's attention.

[0080] Pressure feedback: When the in-vehicle helmet system needs to convey a more urgent signal, pressure changes can be used as a feedback method. For example, the pressure of the helmet increases, prompting the driver to take action when a collision is about to occur; and when the task is completed or the danger is eliminated, the pressure will gradually decrease.

[0081] Temperature feedback: In certain situational tasks (such as high or low temperature environments), temperature changes can be used as an auxiliary feedback method. Temperature changes remind the driver or operator to pay attention to environmental conditions or the working status of vehicle equipment.

[0082] (2) Adjustment of the intensity and frequency of tactile feedback.

[0083] The intensity and frequency of tactile feedback should be adjusted according to the requirements of different tasks to ensure accurate information delivery and balanced cognitive load: Task urgency adjustment: For high-urgency tasks (such as collision warnings, forward obstacle prompts, etc.), the intensity of tactile feedback should be higher and the frequency should be faster to ensure that the driver can respond quickly.

[0084] Task complexity adjustment: During complex tasks (such as long driving or complex navigation), the intensity of tactile feedback should be moderate and the frequency should be low to avoid excessively interfering with the driver's attention and reduce cognitive burden.

[0085] Feedback mode adaptation: In a multi-tasking environment, tactile feedback should be able to automatically adapt to changes in tasks. When the driver's task load increases, the intensity and frequency of tactile feedback can be moderately increased to help allocate cognitive resources; when the task load is low, the intensity and frequency of feedback can be reduced to avoid interference.

[0086] (3) Personalization and adaptability of tactile feedback.

[0087] Different drivers or operators have different tactile sensitivities, so tactile feedback should be personalized and adaptive: Personalized adjustment: The system should allow the driver to adjust the intensity and frequency of tactile feedback according to personal perception preferences. For example, the intensity of tactile feedback can be adjusted according to the driver's needs to ensure that the feedback is within the user's perception range and does not cause discomfort.

[0088] Adaptive adjustment: The system can automatically adjust the haptic feedback mode according to the driver's cognitive load and situational task changes. For long-term operation or high-stress tasks, the system should automatically adjust the frequency and intensity of haptic feedback to help the driver maintain optimal attention and response.

[0089] (4) Environmental adaptability design.

[0090] Haptic feedback should be adaptively adjusted according to changes in the driving environment: Lighting and noise environment: In low-light or strong-light environments, the transmission of visual information may be affected. At this time, tactile feedback should be appropriately enhanced to ensure that the driver can accurately perceive task information. In addition, in a noisy environment, tactile feedback can replace part of the auditory signal to ensure the effective transmission of information.

[0091] Temperature and humidity changes: When the temperature or humidity of the vehicle environment changes, the system can use tactile feedback of temperature changes or pressure changes to remind the driver. For example, when the temperature inside the car is too high, the helmet can use temperature feedback to remind the driver to adjust the environment inside the car.

[0092] (5) Optimization of tactile feedback in multi-tasking environments.

[0093] In a multitasking environment, haptic feedback should be designed in a way that does not cause information overload or interfere with the user’s primary task: Task switching optimization: When switching tasks, tactile feedback should remain consistent and predictable between switching between different tasks to avoid confusion or stress for users due to frequent tactile changes.

[0094] Task priority adjustment: When multiple tasks are executed in parallel, the system should dynamically adjust the distribution of tactile feedback according to the priority of the tasks. For example, for urgent tasks, the system can give priority to strong tactile feedback, while reducing the feedback frequency of other tasks to reduce cognitive burden.

[0095] Multimodal combination design strategy: When it comes to the multimodal interactive feedback design strategy for military vehicle-mounted helmet systems, it is necessary to comprehensively consider the characteristics and advantages of various interactive modes (such as vision, hearing, touch, etc.) to create an interactive experience that is efficient, intuitive, and adaptable to the battlefield environment. The following is a detailed description of the multimodal interactive feedback design strategy: Integrate the advantages of multiple modalities.

[0096] Visual feedback: Use the helmet's display screen to provide rich visual information, such as maps, target indications, status updates, etc. Visual feedback should be clear, easy to understand, and able to maintain readability under different lighting and viewing conditions. Auditory feedback: Deliver key information or attract soldiers' attention through sound signals such as voice prompts and alarm sounds. Auditory feedback should be clear, loud, and able to be effectively recognized in a noisy battlefield environment. Haptic feedback: Use tactile methods such as vibration and pressure to provide soldiers with instant feedback. Haptic feedback can be used to remind soldiers of important information or to interact when they cannot view the screen or hear the sound.

[0097] Design collaborative interactions.

[0098] Ensure that different modes of interaction can work together to provide more complete and accurate feedback. For example, when a soldier issues a command through voice, the system can provide feedback through both visual and auditory means to enhance the communication of information. Design a reasonable mode switching mechanism so that soldiers can quickly switch between different interaction modes as needed. The switching operation should be simple and intuitive, and avoid confusion or misoperation during the switching process.

[0099] Provide instant feedback.

[0100] Multimodal interactive feedback should provide immediate and accurate feedback so that soldiers can understand the operation results and system status in a timely manner. For example, when a soldier issues an instruction, the system should immediately give corresponding feedback to confirm that the instruction has been received and processed. For critical information or warnings, the system should use stronger and more obvious feedback methods (such as highlight display, emergency alarm sound, etc.) to attract the attention of soldiers.

[0101] Design adaptive feedback.

[0102] Consider the interactive feedback requirements under different battlefield environments and mission requirements. For example, at night or in low-light environments, visual feedback may require higher brightness and contrast; on a noisy battlefield, auditory feedback may require louder volume and clearer voice prompts. Design adjustable feedback parameters (such as brightness, volume, vibration intensity, etc.) so that soldiers can adjust them according to personal preferences and battlefield environments.

[0103] Optimize the interaction process.

[0104] Design concise and intuitive interaction processes to reduce soldiers' cognitive burden and operational difficulty. Avoid using complex operation steps or redundant feedback information. Make full use of the advantages of multimodal interaction and provide soldiers with a variety of interaction options. Soldiers can choose the most suitable interaction method according to their own habits and battlefield environment.

[0105] Consider emotional design.

[0106] Incorporate emotional elements into interactive feedback design to enhance soldiers' sense of belonging and self-confidence. For example, use encouraging voice prompts and warm sound effects to make soldiers feel the care and support of the system. Consider the psychological pressure and emotional changes of soldiers on the battlefield, and design feedback methods that can relieve stress and improve morale. For example, give positive feedback and rewards when completing tasks.

[0107] Conduct user testing and continuous optimization.

[0108] Collect feedback from soldiers through user testing to understand their satisfaction with the multimodal interactive feedback design and suggestions for improvement. Continue to optimize the design strategy based on the test results to improve the system's usability and user experience. Pay attention to the actual needs and usage habits of soldiers on the battlefield, and continuously improve and refine the interactive feedback design.

[0109] Multimodal interaction design.

[0110] The multimodal human-computer interaction design of military vehicle-mounted helmet systems needs to be carefully planned to ensure an efficient and safe interactive experience in a complex and ever-changing battlefield environment. In terms of design strategy, we emphasize the integration and complementarity of multimodality, and flexibly configure the interactive mode according to the situational characteristics of the two stages of marching and combat. In the marching stage, low-frequency tactile display information and verbal language information are used to ensure the accuracy and timeliness of information transmission, and a repeated playback or confirmation mechanism of voice commands is designed to deal with possible recording quality or understanding difficulties. In the combat stage, due to the noisy environment and complex tasks, we mainly rely on non-verbal auditory information (such as sound guidance) and visual information (such as instructions on the display screen) to reduce distraction and misoperation, and design the synchronous presentation of sound and visual information to enhance the reliability and accuracy of information. In addition, we strictly control the amount and complexity of interactive information to avoid excessively increasing the user's cognitive load, and reduce cognitive burden by optimizing information presentation methods and interactive processes. Finally, we match the best combination of the three interaction modes of touch, hearing (including verbal and non-verbal) and vision according to the needs of different situational tasks, and design a flexible mode switching mechanism to adapt to the needs of different tasks, so as to provide soldiers with powerful and reliable interaction support on the battlefield, such as Figure 4 shown.

[0111] The reason why the present invention does not use the superposition of three interaction modes is that previous research results have shown that the superposition of three interaction modes may have higher interaction efficiency, but it will also increase the user's mental workload and cognitive load. In the complex task scenarios faced by the vehicle-mounted helmet system, if the user's cognitive load is increased, then this may have a greater negative impact on their performance when performing tasks, leading to misoperation, delayed response, distraction, etc., and excessive use of a single mode to process information will increase the cognitive burden, so the present invention adopts a combination of two modes.

[0112] S5. Conduct subjective and objective evaluation experiments on the multimodal interaction design of the vehicle-mounted helmet system, and verify the effectiveness and practicality of the design strategy by measuring the cognitive load and operational performance of users in response to different interaction feedback in different situational tasks.

[0113] The following is a multi-modal combination experiment exploration from the three sensory channels of vision, hearing, and touch, measuring the differences in task completion time and workload indicators under different modal combinations, based on task performance measurement (task reaction time) and user perception measurement (perceived workload), and using methods such as variance analysis to study the correlation and influence between interaction methods and situational variables, and complete the interaction experiment evaluation of multiple modal combinations under different situational tasks. Finally, based on the evaluation results, the optimal design of the vehicle-mounted helmet system is realized and a solution for the flexible configuration of its interaction modalities is proposed.

[0114] The purpose of the experiment is to test the task performance measurement and user workload under different modal combinations (visual + audio, visual + touch, and auditory + touch). The user perception measurement is mainly evaluated for the combination of interaction modes. Task reaction time refers to the time it takes for participants to respond from receiving feedback information to making a response.

[0115] It is assumed that in task (T1), the combination of interaction modes has a significant impact on the interaction reaction time (F=×××, p=×××), and the average time of the combination of vision + hearing is significantly lower than that of the other three modal combinations. In other words, this interaction mode takes the shortest time to make decisions in task (T1), and the overall performance of the interaction speed is better.

[0116] Experimental subjects.

[0117] This experiment selected 20 people to participate in the test, aged 20 to 30 years old (mean M = 24.35 years, standard deviation SD = 3.42 years), including 10 males and 10 females. All subjects were in good physical condition, right-handed, color blind and color weak, binocular vision or corrected vision, hearing and touch, normal cognitive ability, and the subjects were introduced to the relevant precautions and experimental requirements of the experiment before the experiment to ensure that the subjects had enough sleep the night before. Before the experiment, all subjects were informed of the experimental process and instructions and voluntarily signed a written informed consent.

[0118] Experimental equipment.

[0119] The equipment used in the experiment includes ErgoLAB EEG wearable electroencephalometer, HTC Vive Pro Eye virtual reality eye tracker, ErgoLAB Biosensing smart wearable physiological instrument and ErgoLAB human-computer environment testing cloud platform.

[0120] The experiment is based on the armored vehicle crew task virtual simulation environment designed and built by Unity, and runs in Steam VR. The experimental data recording and collection platform adopts ErgoLAB human-computer environment synchronization platform V3.0. Through the ErgoLAB human-computer environment test cloud platform system, it can synchronously collect and analyze eye movement, physiological, behavioral and other data, and can configure and customize software functions according to its own needs. According to the experimental requirements, the platform is mainly composed of Ergo VR (U3D) VR prototype synchronization module, ErgoLAB Questionnair custom questionnaire scale module, wearable wireless physiological recording module, eye movement tracking recording module, EEG measurement system module, data analysis function module and other main modules through custom configuration, which can realize data synchronization and real-time recording and transmission. Among them, the VR prototype synchronization module simulates and provides the subjects with high-fidelity combat scenes, including mountains, trees, rocks, etc., with normal lighting. The platform provides visual, auditory and tactile prompt information, and the internal data analysis module can record the task performance of the subjects.

[0121] The eye movement indicators were collected using the HTC Vive Pro Eye virtual reality eye tracker, which has a screen resolution of 3K and a sampling frequency of 90Hz, which reduces the screen effect. The sound effects use Hi-Res Audio certification and 3D stereo spatial sound effects, which makes you feel like you are in the scene. During the experiment, the subjects' EDA physiological data and HRV heart rate data were collected synchronously through the ErgoLAB Biosensing smart wearable physiological instrument, with a sampling frequency of 4096Hz / channel and a resolution of 16Bits. In addition, the experiment collected EEG and ERP brain wave analysis data through the ErgoLABEEG wearable electroencephalogram, which is equipped with high-precision components, a resolution of 16bit, and a sampling rate of 1024Hz.

[0122] Evaluation indicators.

[0123] The present invention adopts a combination of subjective and objective evaluation methods, as shown in Table 6.1. Among them: the subjective evaluation index uses the NASA-TLX scale; the objective evaluation index includes the reaction time to information feedback, EEG brain wave data (α / β), EDA skin wave data (SC), and HRV heart rate data (LF, HF, LF / HF). Cognitive and emotional conditions are mainly reflected by collecting EEG brain wave signals, EDA skin wave signals, and HRV heart rate signals.

[0124] Table 1 Experimental evaluation indicators

[0125] Experimental design.

[0126] The interactive evaluation adopted a 3×2 two-factor within-subjects experimental design, including three interaction combinations and two stage tasks (march state task and combat state task).

[0127] Factor 1 is the modality type, which includes the following three types: vision + hearing (V+A); vision + touch (V+T); and hearing + touch (A+T). When the dual-modality interactive feedback is performed, the information prompt direction and the direction in which the enemy car appears in the virtual environment are consistent in space, and the multi-modal prompt information is fed back simultaneously.

[0128] Factor 2 is the task type. Task 1 in T1 is in the marching state (reconnaissance, maneuvering). According to the multimodal prompt, after the "current speed is too fast, please slow down" appears in the marching situation, the acceleration and deceleration command task is completed; Task 2 in T2 is in the combat state (attack and warning). According to the multimodal prompt, the command tasks of attacking enemy vehicles and protecting retreat are completed in the combat situation. The task complexity and interactive cognitive load of Task 2 are significantly higher than those of Task 1.

[0129] We tested the subjects in a strictly controlled environment. The interactive space of the test site was precisely set to a size of 400cm×400cm to ensure that there was enough space for various experimental operations. In the Unity test environment, we carefully set the lighting and noise conditions according to the different stages of the task. For T1 Task 1 (marching state), we simulated the environment of marching during the day, using sufficient external light sources with a color temperature of 2700k, while keeping the environment quiet or with only weak noise (40dB-60dB). In T2 Task 2 (combat state), we simulated the combat environment at night or in bad weather, using weak light sources and adding strong noise (90dB~120dB) to simulate a more tense and exciting combat atmosphere. The setting of environmental conditions is intended to more realistically reflect the environmental challenges that armored vehicles may encounter at different combat stages, so as to more accurately evaluate the interactive performance and combat adaptability of the subjects.

[0130] Experimental process: The experiment includes two stages: pre-experiment and formal experiment. In the pre-experiment stage, the subjects are required to be fully familiar with the experimental scene and task operation process, familiar with the specific meaning of visual, auditory and tactile warning information, sign the informed consent form, and conduct pre-experimental exercises. In the formal experimental stage, for example, two tasks need to be completed in the visual + auditory modality (V+A). Each time the "Menu" key in the handle is pressed, it is recorded as a reaction time. The experimenter will use the ErgoLAB human-computer environment test cloud platform system to record the completion time of the subjects. When the modal feedback task experiment of group a is completed, it will switch to the two tasks under the visual + tactile modality (V+T) of group b, and the test environment and equipment status will be reset to the initial state. Repeat the steps of the previous test until modalities a, b, and c have completed two tasks. The specific steps are shown in Table 2: Table 2 Task experiment design table

[0131] (1) Experimental process under combat status: ①Vision + Auditory (V+A) 5 seconds after entering the virtual combat environment, an enemy target will appear (in Unity, the enemy tank slowly drives towards us and moves to a certain distance), accompanied by an auditory danger prompt sound (A) + visual warning dynamic information (V). The subject needs to respond to the enemy target. When the enemy tank moves to a position suitable for firing, click the handle button to complete Task 1. After completing Task 1, 5 seconds later (triggered by being attacked), the virtual combat environment will simultaneously appear a visual warning prompt dynamic information (V) + warning prompt sound (A) of "engine damage". The subject needs to respond to the warning prompt and click the handle button to complete the task.

[0132] ②Vision + Touch (V+T) 5 seconds after entering the virtual combat environment, an enemy target will appear (the enemy tank in Unity slowly drives towards us and moves to a certain distance) accompanied by handle vibration (tactile) + visual warning dynamic information (UI aiming interface flicks), and the subject needs to respond to the enemy target. When the enemy tank moves to a position suitable for firing, click the handle button to complete Task 1. After completing Task 1, after 5 seconds, the virtual combat environment will simultaneously appear a visual warning prompt dynamic information of "engine damage" + handle vibration (tactile), and the subject needs to respond to the warning prompt and click the handle button to complete the task.

[0133] ③Hearing + Touch (A+T) 5 seconds after entering the virtual combat environment, an enemy target will appear (the enemy tank in Unity slowly drives towards us and moves to a certain distance), accompanied by handle vibration (tactile) + auditory danger warning sound (audio). The subjects need to respond to the enemy target. When the enemy tank moves to a position suitable for firing, click the handle button to complete Task 1. After completing Task 1, after 5 seconds, the auditory danger warning sound (audio) + handle vibration (tactile) of "engine damage" will appear in the virtual combat environment at the same time. The subjects need to respond to the warning prompt and click the handle button to complete the task.

[0134] After completing the situational tasks in each modality, the subjects were required to fill in the NASA scale.

[0135] (2) Experimental process under marching state: ①Vision + Touch (V+T) Enter the virtual reality environment. Five seconds after entering the virtual combat environment, the visual information "maintain a constant speed" and the handle vibration will appear simultaneously. The subjects need to respond to the task. When the driving speed is transferred to a suitable state (for example, the current speed is too fast, and the instruction is received in the acceleration state, and the engine needs to be decelerated to a constant speed state), click the handle button to complete the task.

[0136] ②Vision + Auditory (V+A) Enter the virtual reality environment. Five seconds after entering the virtual combat environment, the visual information + prompt sound (audio) of "maintaining a constant speed" will appear simultaneously. The subjects need to respond to the task. When the driving speed is transferred to a suitable state (for example, the current speed is too fast, and the command is received in the acceleration state, and the engine needs to be decelerated to a constant speed state), click the handle button to complete the task.

[0137] ③Hearing + Touch (A+T) Enter the virtual reality environment. 5 seconds after entering the virtual combat environment, a prompt sound (audio) + handle vibration of "maintaining a constant speed" will appear at the same time. The subjects need to respond to the task. When the driving speed is transferred to a suitable state (for example, the current speed is too fast, and the command is received in the acceleration state, and the engine needs to be decelerated to a constant speed state), click the handle button to complete the task.

[0138] After completing the situational tasks in each modality, the subjects were required to fill in the NASA scale.

[0139] Experimental data processing and analysis The test data will be analyzed by IBM SPSS 27. The normality of task performance measurement and user perception data was verified by Shapiro-Wilk (p>0.05). Because the experimental independent variable is an intra-subject variable and the dependent variable is a continuous variable, the experimental data is analyzed by repeated measures variance analysis. Repeated measures variance analysis is used to detect the impact of multimodal combination methods on interaction effectiveness in different situational tasks and to compare between groups.

[0140] (1) Task response time As shown in Tables 3 and 4, the data analysis results show that the interaction combination has a significant effect on the completion time of the interaction task in both the combat task (T1) and the marching task (T2) (p < 0.05). 20 subjects were invited to participate in this experiment. Each subject was measured for the task reaction time under the V+A interaction mode (M = 1.357, SD = 0.783), V+T interaction mode (M = 1.897, SD = 1.046), and A+T interaction mode (M = 1.311, SD = 0.828) in the attack task of the combat phase T1, as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω = 0.638, and there was a significant difference in the task reaction time of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F (1.907, 36.237) = 4.037, p = 0.028, partial η2 = 0.175. The average time of the combination of hearing and touch was significantly lower than that of the other two modal combinations. (SD=1.046). In the T1 combat phase warning task, the task reaction time was measured in the V+A interaction mode (M=0.889, SD=0.350), V+T interaction mode (M=1.339, SD=0.908), and A+T interaction mode (M=1.154, SD=0.640), as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.747, and there was a significant difference in the task reaction time of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.494, 28.385)=2.969, p=0.007, partial η2=0.135. The average time of the combination of vision and hearing is significantly lower than that of the other two modal combinations. In the two tasks in the combat phase, the task reaction time of the visual and tactile modalities is significantly higher than that of the other two groups of interaction combinations. Therefore, the overall standard deviation is relatively large, such as Figure 5 shown.

[0141] Table 3 The effect of interaction mode on task reaction time (s) in T1 situation task

[0142] Each participant's task reaction time was measured in the contextual task of the T2 marching phase under the V+A interaction mode (M=1.609, SD=0.939), V+T interaction mode (M=1.791, SD=1.448), and A+T interaction mode (M=0.898, SD=0.508). The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.783, and there was a significant difference in the task reaction time of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.567, 29.767)=6.268, p=0.009, partial η2=0.248. The average time of the combination of auditory and tactile modes was significantly lower than that of the other two modal combinations, and the task reaction time of the visual and tactile modalities was significantly higher than that of the other two groups of interaction combinations, so the overall standard deviation was large (SD=1.448).

[0143] Table 4 The influence of interaction mode on task reaction time (s) in T2 situation task

[0144] (2) Perceived workload As shown in Table 5, the NASA Task Load Index (NASA-TLX) showed that the interaction mode had a significant effect on the effort level in the T1 warning task (F=6.552, p=0.005), but had no significant effect on the other indicators. During the T1 attack task, the average NASA-TLX score of the subjects in the auditory + tactile mode was 29.3 (SD=3.58), which was significantly higher than the other two postures and had a smaller standard deviation. Therefore, it can be judged that the perceived workload in the auditory + tactile mode was the highest. No significant effect was found in the interaction mode of the T1 attack task on any indicator of perceived workload. In terms of the combination of interaction modes, the average score of the interaction mode combining tactile and auditory was 26.7 (SD=4.08), and the average score of the interaction mode combining visual and auditory was 24.1 (SD=4.12), and all evaluation indicators were the lowest values. Therefore, the interaction mode combining visual and auditory may be the interaction combination with the lowest perceived workload.

[0145] Table 5 The impact of interaction mode on perceived workload under T1 / T2 tasks

[0146] The interaction mode of the T2 task had no significant effect on any of the indicators of perceived workload. In terms of the interaction mode combination, the average score of the interaction mode combining hearing and touch was 16.0 (SD=3.08) and all evaluation indicators were the lowest. Therefore, the interaction mode combining hearing and touch may be the interaction combination with the lowest perceived workload.

[0147] (3) EEG index measurement results Calculating concentration and relaxation is usually achieved by analyzing specific frequency bands in electroencephalogram (EEG) signals. EEG activity is closely related to an individual's perception, memory, emotion, language, action decision-making, and other high-level cognitive processes. It can reflect the activity state of different areas of the cerebral cortex and can be used to monitor a person's physiological and psychological state and the interaction between different functional areas of the brain. The signals recorded by EEG devices are divided into multiple frequency bands, including α, β, θ, and δ waves. These brain waves play a key role in different physiological and psychological states. In this experiment, the α / β index is mainly examined. The higher the α / β value, the higher the user's cognitive and emotional arousal, and it also means that the user is more excited and nervous during the operation.

[0148] The ErgoLAB human-computer environment synchronization cloud platform is used to synchronously collect and analyze the EEG data and physiological data of the subjects when they complete the task. The ErgoLAB EEG analysis module analyzes the EEG data of the subjects when they complete different situational tasks in different modal combinations. The indicator of EEG data analysis is the α / β of the subject's task completion stage.

[0149] As shown in Tables 6 and 7, the data analysis results show that the interaction combination mode has a significant effect on the α / β value of the interaction task in both the combat task (T1) and the marching task (T2) (p < 0.05). 20 subjects were invited to participate in this experiment. Each subject was measured in the T2 marching phase task in the V+A interaction mode (M=1.535, SD=0.644), V+T interaction mode (M=1.042, SD=0.716), A+T interaction mode (M=1.301, SD=0.475) α / β value, cognitive load, as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω = 0.996, and there was a significant difference in the α / β values ​​of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F (1.993, 37.863) = 5.175, p = 0.010, partial η2 = 0.214. The α / β value of the combination of vision and touch was significantly lower than that of the other two modal combinations, such as Figure 7 As shown. (SD=1.042).

[0150] The task reaction time of each subject in the attack task of the T1 combat phase was measured in the V+A interaction mode (M=1.093, SD=0.696), V+T interaction mode (M=1.935, SD=1.670), and A+T interaction mode (M=1.201, SD=0.322), as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.921, and there was a significant difference in the α / β values ​​of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.842, 34.997)=17.917, p<0.001, partial η2=0.485. The average time of the combination of vision and touch was significantly higher than that of the other two modes. (SD=1.935).

[0151] In the warning task at the T1 combat stage, the task reaction time was measured in the V+A interaction mode (M=1.833, SD=0.575), V+T interaction mode (M=1.097, SD=0.682), and A+T interaction mode (M=1.951, SD=0.581). The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.162, and there was a significant difference in the task reaction time of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.690, 32.114)=12.441, p<0.001, partial η2=0.396. The average time of the combination of vision and touch was significantly lower than that of the other two modal combinations.

[0152] From the results, we can see that in the T1 combat attack task, the subjects' cognitive attention and emotional arousal were higher in the V+T combination mode, and they were more likely to feel nervous and excited, while the α / β value of V+A was the lowest, indicating that the visual + auditory feedback mode produced less tension and less psychological load. Figure 6 shown.

[0153] Table 6 The influence of interaction mode on α / β value in T1 situation task

[0154] Table 7 The influence of interaction mode on α / β value in T2 situation task

[0155] (4) HRV / EDA index measurement results The skin electrical index analyzed in this experiment is SC, and the heart rate variability HRV index is LF / HF. HRV heart rate signal is the data source for analyzing heart rate variability. It can quantitatively evaluate the tension and balance of cardiac sympathetic and vagus nerve activities and their impact on cardiovascular system activities. Its related index LF / HF can effectively reflect cognitive load. The increase in cognitive load can cause a decrease in the index reflecting parasympathetic nerve activity. At the same time, the skin electrical data index SC was analyzed. Skin electrical response is one of the most sensitive emotional feedbacks. It is caused by the autonomous activation of skin sweat glands and is closely related to emotions, arousal, and attention. SC indicates the degree of physiological arousal of the human body. When the degree of physiological arousal is higher, the corresponding SC index is also higher, indicating that the user has made a higher cognitive effort.

[0156] Table 8 The effect of interaction modality on SC (μS) in T1 contextual task

[0157] As shown in Tables 8 and 9, the data analysis results show that the interaction combination mode has a significant effect on the SC of the interaction task in both the combat task (T1) and the marching task (T2) (p < 0.05). 20 subjects were invited to participate in this experiment. Each subject was measured for task reaction time in the attack task of the combat phase T1 under the V+A interaction mode (M = 8.198, SD = 3.581), V+T interaction mode (M = 9.458, SD = 3.845), and A+T interaction mode (M = 8.690, SD = 3.696). The Greenhaus-Geisler estimate of the deviation from sphericity was ω = 0.718, and there was a significant difference in the α / β values ​​of the V+A interaction mode / V+T interaction mode / A+T interaction mode, F (1.435, 27.273) = 20.261, p < 0.001, partial η2 = 0.516. The SC of the combination of vision and hearing was significantly lower than that of the other two modal combinations (SD=3.581).

[0158] In the warning task at the T1 combat stage, the task reaction time was measured in the V+A interaction mode (M=8.905, SD=4.381), V+T interaction mode (M=9.731, SD=4.535), and A+T interaction mode (M=8.867, SD=4.111), as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.961, and there was a significant difference in SC between the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.923, 36.536)=6.343, p=0.005, partial η2=0.250. The average SC of the combination of vision and touch was significantly higher than that of the other two modes.

[0159] Table 9 The effect of interaction modality on SC (μS) in T2 context task

[0160] Each subject measured the SC value and cognitive load in the V+A interaction mode (M= 9.378, SD=3.336), V+T interaction mode (M=9.848, SD=3.319), and A+T interaction mode (M=8.799, SD=3.228) during the T2 marching phase task, as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.950, and there was a significant difference in SC between the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.901, 36.113)=9.971, p<0.001, partial η2=0.344. The SC value of the combination of hearing and touch was significantly lower than that of the other two modal combinations, such as Figure 8 As shown. (SD=3.228).

[0161] The HRV index of heart rate variability is LF / HF, as shown in Table 10 and Table 11. The data analysis results show that the interaction combination mode has a significant effect on the LF / HF of the interaction task in both the combat task (T1) and the marching task (T2) (p < 0.05). 20 subjects were invited to participate in this experiment. Each subject was measured in the attack task of the combat stage T1 under the V+A interaction mode (M=0.830, SD=0.278), V+T interaction mode (M=1.586, SD=0.631), and A+T interaction mode (M=1.321, SD=0.610) task reaction time, as shown in the table. The Greenhaus-Geisler estimate of desphericity was ω = 0.782, and there was a significant difference in LF / HF between the V+A interaction mode / V+T interaction mode / A+T interaction mode, F (1.563, 29.701) = 17.399, p < 0.001, partial η2 = 0.478. The LF / HF of the combination of vision and hearing was significantly lower than that of the other two modal combinations (SD = 0.278).

[0162] In the warning task at the T1 combat stage, the task reaction time was measured in the V+A interaction mode (M=1.332, SD=0.637), V+T interaction mode (M=1.569, SD=0.640), and A+T interaction mode (M=0.901, SD=0.285), as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.719, and there was a significant difference in LF / HF between the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.438, 27.329)=13.483, p<0.001, partial η2=0.415. The average LF / HF of the combination of hearing and touch was significantly lower than that of the other two modal combinations, such as Fig. 9 shown.

[0163] Table 10 The effect of interaction modality on LF / HF in T1 contextual task

[0164] Each subject was measured for LF / HF values ​​and cognitive load in the T2 marching phase task under the V+A interaction mode (M=1.580, SD=0.633), V+T interaction mode (M=1.309, SD=0.623), and A+T interaction mode (M=0.837, SD=0.290), as shown in the table. The Greenhaus-Geisler estimate of the deviation from sphericity was ω=0.781, and there was a significant difference in LF / HF between the V+A interaction mode / V+T interaction mode / A+T interaction mode, F(1.562, 29.669)=16.201, p<0.001, partial η2=0.460. The LF / HF value of the combination of hearing and touch was significantly lower than that of the other two modal combinations, such as Fig.10 shown.

[0165] Table 11 The influence of interaction modality on LF / HF in T2 context task

[0166] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0167] Therefore, the present invention adopts the above-mentioned multi-modal interactive vehicle-mounted helmet system design method to enhance the information processing capability and combat effectiveness of military crew members in complex battlefield environments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for designing a multimodal interactive vehicle-mounted helmet system, characterized in that: The following steps are involved: Step S1: considering the information processing process of the occupants in different situational tasks, considering the cognitive load and resource allocation issues, and designing the human-computer interaction design principle of the vehicle helmet to balance cognitive resources; Step S2, formulating a multimodal design strategy for a vehicle helmet system based on context awareness based on the vehicle helmet human-computer interaction design principle of balanced cognitive resources in step S1; Step S3: designing a context-aware human-computer interaction system based on the context-aware vehicle-mounted helmet system multimodal design strategy of step S2.

2. The method for designing a multimodal interactive vehicle helmet system according to claim 1, characterized in that: In step S1, different situational tasks include marching phase tasks and combat phase tasks, and combat phase tasks include attack tasks and warning tasks.

3. The method for designing a multimodal interactive vehicle helmet system according to claim 2, characterized in that: In step S1, the design principle of vehicle-mounted helmet human-computer interaction for balancing cognitive resources is: Excessive use of a single channel leads to an imbalanced cognitive load. In this case, other channels should be introduced to achieve resource redistribution. Excessive use of multiple channels leads to information overload. At this time, reduce the number of channels to achieve a balanced state of cognitive load.

4. The method for designing a multimodal interactive vehicle helmet system according to claim 3, characterized in that: Channels include visual channels, auditory channels, and tactile channels.

5. The method for designing a multimodal interactive vehicle helmet system according to claim 4, characterized in that: In step S2, a multimodal design strategy of a vehicle-mounted helmet system based on contextual awareness includes a visual modal design strategy, an auditory modal design strategy, a tactile modal design strategy, and a multimodal combination design strategy; Among them, the visual modality design strategy includes the structure layer, framework layer and presentation layer. The structure layer defines the information architecture based on the associated attributes of information, the display type of information, and the operation process of the task; the framework layer is modularly designed based on visual laws and perceptual processing; the presentation layer is designed based on visual unity, environmental adaptability and semantic rationality; Auditory modality design strategies include clarity and loudness control, noise reduction capabilities, directional audio delivery, context-aware audio cues, communications, and command and control capabilities; Tactile modality design strategies include vibration feedback, temperature perception, pressure perception, tactile interface, and personalized adaptation; The multimodal combination design strategies include the first combination strategy, the second combination strategy and the third combination strategy. The first combination strategy is a combination of the visual modal design strategy and the auditory modal design strategy, the second combination strategy is a combination of the visual modal design strategy and the tactile modal design strategy, and the third combination strategy is a combination of the auditory modal design strategy and the tactile modal design strategy.

6. The method for designing a multimodal interactive vehicle helmet system according to claim 5, characterized in that: In step S3, the context-aware human-computer interaction system includes a task context setting module, a multimodal design strategy module, and a feedback and evaluation module; The mission scenario setting module is used to set marching missions, attack missions and warning missions; The multimodal design strategy module is used to set a first combination strategy, a second combination strategy and a third combination strategy; The feedback and evaluation module is used to monitor and evaluate the effectiveness of the interactive system in real time based on the task context and multimodal design strategy.

Images