A multi-mode azimuth warning configuration method and system based on situation awareness

Abstract

The application belongs to the technical field of human-computer interaction, and discloses a multi-modal orientation alarm configuration method and system based on situation awareness, which comprises the following steps: obtaining multi-source sensor data, deriving threat three-dimensional coordinates of a target object, and generating an orientation alarm constraint set; establishing a task resource occupation characteristic vector based on type information; constructing a physiological feature set based on physiological signals; calculating an attention level index and a cognitive load level index based on the physiological feature set, and constructing a situation awareness vector; outputting an optimal alarm mode and triggering alarm information; the application realizes adaptive selection and updating of an orientation alarm channel in a multi-task operation environment by jointly incorporating task resource occupation characteristics and operator situation awareness into alarm mode applicability evaluation and configuration decision, thereby reducing orientation recognition error rate and reaction time delay, reducing interference of the alarm on a main task, and improving usability and interaction efficiency of the alarm system in a complex operation scenario.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically, to a multimodal orientation alarm configuration method and system based on situational awareness. Background Technology

[0002] In multi-tasking operating environments such as vehicles and industrial control systems, operators typically need to perceive and respond to alarm information from different directions while continuously performing their primary tasks to support timely decision-making and operations. Such alarm information often has a clear directional attribute, and its transmission efficiency directly affects the operator's understanding of the external situation and the accuracy of subsequent actions. Therefore, how to efficiently and reliably transmit directional alarm information to operators in complex operating scenarios has become a crucial issue in the design of human-computer interaction systems.

[0003] Currently, alarm systems primarily use fixed alarm channels or rule-based alarms, as well as multi-channel, multi-modal alarms. Alarms based on fixed channels or rule-based alarms are typically experience-driven and don't fully consider the differences in resource consumption characteristics across various tasks. In multi-task environments, this can lead to mismatches between alarm modalities and the main task's resource requirements, reducing the efficiency of location alarm recognition. Alarms based on multi-channel, multi-modal alarms focus on improving the presentation or prompting of alarm information. They are usually designed based on single-scenario or individual experience, lacking a quantitative evaluation model that integrates task characteristics and situational awareness to assess the applicability of alarm channels. This results in a lack of adaptive modal configuration capabilities in complex and changing scenarios.

[0004] In view of this, the present invention proposes a multimodal orientation alarm configuration method and system based on situational awareness to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and achieve the above objectives, the present invention provides the following technical solution: A multimodal azimuth alarm configuration method based on situational awareness, the method comprising: S1. Deploy an external sensing sensor group on the operating platform to scan the surrounding environment of the target object in real time, acquire multi-source sensor data, analyze and calculate the multi-source sensor data, obtain the threat three-dimensional coordinates of the target object, and generate a directional alarm constraint set D based on the threat three-dimensional coordinates. S2. Obtain the type information of the current operation task, calculate the channel occupancy index of the current operation task based on the type information, and establish the task resource occupancy feature vector T based on each channel occupancy index; S3. Acquire the physiological signals of the operator, preprocess the physiological signals and extract features to obtain physiological features, and construct a set of physiological features P based on the physiological features; S4. Calculate the attention level index and cognitive load level index based on the physiological feature set P, and construct the situational cognition vector C based on the attention level index and cognitive load level index. S5. Determine the candidate alarm mode set M. Using the task resource occupancy feature vector T, situational awareness vector C, and azimuth alarm constraint set D as constraints, calculate the applicability evaluation value of each alarm mode in the candidate alarm mode set M for azimuth information transmission. Based on the applicability evaluation value and the decision variables of each alarm mode, output the optimal alarm mode and trigger the alarm information.

[0006] Furthermore, the steps for generating the azimuth alarm constraint set D are as follows: S11. Multi-source sensor data includes observation data and the current attitude information of the operating platform. Each external sensing sensor outputs observation data of a target object. The observation data of each external sensing sensor is converted into a three-dimensional position measurement vector in the world coordinate system. Based on the observation uncertainty of each external sensing sensor, the inverse matrix of the measurement covariance matrix of each external sensing sensor is constructed to obtain the observation quality of each external sensing sensor. The observation quality, the inverse matrix of the measurement covariance matrix, and the three-dimensional position measurement vector are fused to calculate the threat three-dimensional coordinates of the target object. ; S12. Obtain the current attitude information of the operating platform, based on the threat 3D coordinates of the target object. Based on its own posture information, a relative coordinate system with the operator as the origin is established, and the relative pose of the target object relative to the operator is calculated. Within the relative coordinate system, the horizontal 360° azimuth plane centered on the operator is denoted as the overall azimuth region. The overall azimuth region is divided equally into eight directions according to angle: front, back, left, right, left front, right front, left back, and right back. The eight directions are encoded separately, and an azimuth alarm constraint set D∈{1,2,3,…,8} is generated based on the relative pose. The numerical values ​​of each element in the azimuth alarm constraint set D are matched with the corresponding azimuth.

[0007] Furthermore, calculate the threat three-dimensional coordinates of the target object. The method is as follows:

[0008] in, For the first The threat's three-dimensional coordinates obtained from the secondary fusion For the first During the second fusion Normalized fusion weights assigned to observation data from external sensing sensors. For the first The external sensing sensor in the first The three-dimensional position measurement vector after coordinate unification during the second fusion. To and The inverse of the corresponding measurement covariance matrix, For the first The external sensing sensor in the first Observation quality during secondary fusion For the first The reliability coefficient of an external sensing sensor. This represents the total number of external sensing sensors.

[0009] Furthermore, the method for establishing the task resource consumption feature vector T is as follows: Type information includes task type, operation frequency, interface interaction quantity, and control input vector. Based on task type, operation frequency, interface interaction quantity, and control input vector, the interaction frequency is calculated within any time window Δt. Hearing occupancy ratio and control input intensity The data was then normalized to quantify the visual channel occupancy index. Auditory channel occupancy index and control channel occupancy indicators Map the occupancy indicators of each channel to a unified interval [0,1] to form a task resource occupancy feature vector. .

[0010] Furthermore, calculate the interaction frequency. Hearing occupancy ratio and control input intensity The method is as follows: ; in, At the current assessment point, This represents the number of interface interactions within the time window. The cumulative duration of auditory output within the time window. For a moment Continuous control input vector; Visual channel occupancy index Auditory channel occupancy index and control channel occupancy indicators The quantification method is as follows:

[0011] in, , , These represent the task baseline occupancy rates for the visual, auditory, and control channels, respectively. , , These are the weighting coefficients for the visual channel, auditory channel, and control channel, respectively. as well as Interaction frequency The extreme values, as well as For auditory occupancy ratio The extreme values, as well as To control input intensity The extreme values, It is a preset minimum positive number.

[0012] Furthermore, the method for constructing the situation awareness vector C is as follows: Based on physiological feature set Output attention level index The calculation model is as follows:

[0013] in, This represents the weight vector of the attention model. For the bias term of the attention model, This is the dimensionless eigenvector after baseline correction. For a set of physiological features The feature vector extracted and concatenated within the current time window Feature vector of the operator in the baseline state The mean vector in each dimension, Feature vector of the operator in the baseline state In the standard deviation vector of each dimension, It is a preset minimum positive number; Based on physiological feature set Output cognitive load level indicators The calculation model is as follows:

[0014] in, This is the weight vector of the load model. This is the bias term for the load model. This is a truncation function; Cognitive load level index and attention level indicators By fusing the data, a situational awareness vector is obtained. .

[0015] Furthermore, the method for calculating the applicability evaluation value is as follows: The candidate alarm modal set is ,in Indicates visual alarm modality, Indicates auditory alarm mode, Representing tactile alarm modality; [This refers to] the task resource usage feature vector. Situational cognition vector The evaluation input vector X is obtained by normalizing and combining the azimuth alarm constraint set D; based on the evaluation input vector... Establish an optimization model with the objective of maximizing the applicability evaluation value of directional information transmission; based on the optimization model, analyze the candidate alarm mode set. The applicability evaluation value for each alarm mode is calculated and output. The applicability evaluation value model is as follows:

[0016] in, For alarm mode The applicability evaluation value of location information transmission under given conditions. , , Alarm modes The incremental resource usage of the visual, auditory, and control channels. For alarm mode Reliability in orientation D.

[0017] Furthermore, the method for outputting the optimal alarm mode is as follows: The optimization model is:

[0018] in, This indicates whether to select the alarm mode. Decision variables, , , These are the safety margins for the visual, auditory, and control channels, respectively. For alarm mode Feasibility indicators; The evaluation values ​​of each alarm mode are obtained through the optimization model, and the alarm mode with the largest evaluation value is selected as the optimal alarm mode for output.

[0019] Furthermore, safety margins for visual, auditory, and control channels. , , The method for obtaining it is as follows: The general method for adjusting the safety margin is as follows:

[0020] in, For channel System latency, For channel Maximum delay, For channel The initial value of the safety margin. For channel Evaluation window Indicates the visual channel. Indicates the auditory channel. Indicates the control channel.

[0021] A multimodal azimuth alarm configuration system based on situational awareness, the system comprising: The orientation alarm constraint set generation module is used to deploy an external sensing sensor group on the operating platform to scan the surrounding environment of the target object in real time, acquire multi-source sensor data, analyze and calculate the multi-source sensor data, obtain the threat three-dimensional coordinates of the target object, and generate the orientation alarm constraint set D based on the threat three-dimensional coordinates. The task resource acquisition and evaluation module is used to acquire the type information of the current operation task, calculate the channel occupancy index of the current operation task based on the type information, and establish a task resource occupancy feature vector T based on each channel occupancy index. The physiological feature extraction module is used to acquire the operator's physiological signals, preprocess the physiological signals and extract features to obtain physiological features, and construct a physiological feature set P based on the physiological features; The situational awareness assessment module is used to calculate attention level indicators and cognitive load level indicators based on the physiological feature set P, and to construct a situational awareness vector C based on the attention level indicators and cognitive load level indicators. The multimodal alarm decision module is used to determine the candidate alarm mode set M. With the task resource occupancy feature vector T, situational awareness vector C, and azimuth alarm constraint set D as constraints, it calculates the applicability evaluation value of each alarm mode in the candidate alarm mode set M for the transmission of azimuth information. Based on the applicability evaluation value and the decision variables of each alarm mode, it outputs the optimal alarm mode and triggers alarm information.

[0022] The technical effects and advantages of the multimodal azimuth alarm configuration method and system based on situational awareness of this invention are as follows: This invention incorporates task resource occupancy characteristics and operator situational awareness into alarm modality applicability assessment and configuration decision-making, thereby enabling adaptive selection and updating of directional alarm channels in multi-task operation environments. This reduces the directional identification error rate and response latency, minimizes alarm interference with the main task, and improves the availability and interaction efficiency of the alarm system in complex operation scenarios. This invention calculates attention level and cognitive load level indicators based on real-time physiological signals, and combines them with the resource consumption characteristics of the current task to quantitatively evaluate the channel applicability of visual, auditory, and tactile modalities. This enables the system to dynamically adjust the alarm modal configuration according to fluctuations in cognitive load, effectively reducing interference with the main task while ensuring the efficiency of location information transmission, reducing misjudgment, and significantly shortening reaction time. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method steps of the present invention; Figure 2 This is a flowchart illustrating the overall solution of the present invention; Figure 3 This is a system module block diagram of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In one embodiment, a multimodal azimuth alarm configuration method based on situational awareness is disclosed, such as... Figure 1 As shown, it includes: S1. Deploy an external sensing sensor group on the operating platform to scan the surrounding environment of the target object in real time, acquire multi-source sensor data, analyze and calculate the multi-source sensor data, obtain the threat three-dimensional coordinates of the target object, and generate a directional alarm constraint set D based on the threat three-dimensional coordinates. S2. Obtain the type information of the current operation task, calculate the channel occupancy index of the current operation task based on the type information, and establish the task resource occupancy feature vector T based on each channel occupancy index; S3. Acquire the physiological signals of the operator, preprocess the physiological signals and extract features to obtain physiological features, and construct a set of physiological features P based on the physiological features; S4. Calculate the attention level index and cognitive load level index based on the physiological feature set P, and construct the situational cognition vector C based on the attention level index and cognitive load level index. S5. Determine the candidate alarm mode set M. Using the task resource occupancy feature vector T, situational awareness vector C, and azimuth alarm constraint set D as constraints, calculate the applicability evaluation value of each alarm mode in the candidate alarm mode set M for azimuth information transmission. Based on the applicability evaluation value and the decision variables of each alarm mode, output the optimal alarm mode and trigger the alarm information.

[0026] Through the above solutions, such as Figure 1 , Figure 2 As shown, this application first utilizes an external sensing sensor array configured on the operating platform to perform real-time scanning of the surrounding environment. These external sensing sensors can be LiDAR, ultrasonic sensors, industrial vision cameras, etc., to acquire multi-source sensor data. This multi-source sensor data includes observation data and the operating platform's current attitude information. Based on the acquired observation data from each external sensing sensor, the coordinate systems of each sensor are transformed to a world coordinate system based on the control panel. Then, a weighted information fusion algorithm, combined with sensor type reliability, is used to calculate the threat's three-dimensional coordinates, thereby accurately locating the target's spatial physical position. Finally, this is combined with the robotic arm's operating platform... The platform's own posture (such as various angles of the control panel and the operator's sitting posture reference) is used to establish a relative coordinate system with the operator as the origin. The horizontal 360° direction is equally divided into eight regions: front, back, left, right, left front, right front, left back, and right back. Each region is assigned a unique numerical identifier (1-8), thereby generating a directional alarm constraint set D. The directional alarm constraint set D serves as the core environmental constraint condition for subsequent alarm modal decision-making, clarifying the direction to which the alarm should point. Then, the type information of the current operation task is obtained, and the channel occupancy index of the current operation task is calculated based on the type information. Based on the channel occupancy index, a task resource occupancy feature vector T is established, which can be calculated by the interaction frequency. Hearing occupancy ratio and control input intensity And quantify it to obtain the visual channel occupancy index. Auditory channel occupancy index and control channel occupancy indicators This process establishes a task resource occupancy feature vector. This vector accurately quantifies the occupancy status of the current control task on the operator's visual, auditory, and control sensory channels, providing a standardized task constraint basis for subsequent alarm modality decision-making. The process involves acquiring the operator's physiological signals, preprocessing and extracting features to obtain physiological characteristics, and constructing a physiological feature set P based on these characteristics. Physiological signals include EEG, ECG / heart rate, and eye movement signals. These signals undergo denoising, artifact removal, resampling, segmentation, and baseline correction before feature extraction to obtain representative feature parameters. These parameters include EEG frequency band power or ratio characteristics, heart rate variability characteristics, fixation, saccades, and pupillary eye movement characteristics. Finally, a physiological feature set P is constructed based on these physiological characteristics. The system calculates attention level and cognitive load indices based on a set P, and constructs a situational awareness vector C based on these indices. The attention level index is a core parameter quantified from physiological characteristics, accurately representing the operator's current level of concentration and engagement with the task, environment, or key information. The cognitive load index, on the other hand, is a core parameter quantified from standardized physiological characteristics, accurately representing the cognitive burden borne by the operator's brain during information processing and decision-making under current task execution and environmental stimuli. Therefore, combining these two indices yields the situational awareness vector C, which represents the operator's overall cognitive state and cognitive resource suitability, thereby accurately assessing the operator's cognitive risk. Finally, a candidate alarm modality set M is determined. ,in Indicates visual alarm modality, Indicates auditory alarm mode, Representing tactile alarm modes; using the task resource occupancy feature vector T, situation awareness vector C, and orientation alarm constraint set D as constraints, calculate the applicability evaluation value of each alarm mode in the candidate alarm mode set M for orientation information transmission. The applicability evaluation value indicates the alarm mode's suitability under the current task resource occupancy, operator situation awareness, and target orientation constraints. The overall effectiveness and adaptability of transmitting location information are considered. A value closer to 1 indicates that the mode effectively transmits location information in the current scenario and has higher adaptability. An optimization model is constructed, which outputs the optimal alarm mode based on the applicability evaluation value and the decision variables of each alarm mode, satisfying the following condition. The alarm mode is selected as the optimal alarm mode, and the corresponding directional alarm information is triggered to complete the configuration of the alarm mode. This invention incorporates the characteristics of task resource consumption and the operator's situational awareness into the alarm mode applicability assessment and configuration decision, thereby realizing the adaptive selection and updating of the directional alarm channel in a multi-task operation environment. This reduces the directional identification error rate and response latency, reduces the interference of alarms on the main task, and improves the availability and interaction efficiency of the alarm system in complex operation scenarios.

[0027] The method for generating the orientation alarm constraint set D consists of the following steps: S11. Multi-source sensor data includes observation data and the current attitude information of the operating platform. Each external sensing sensor outputs observation data of a target object. The observation data of each external sensing sensor is converted into a three-dimensional position measurement vector in the world coordinate system. Based on the observation uncertainty of each external sensing sensor, the inverse matrix of the measurement covariance matrix of each external sensing sensor is constructed to obtain the observation quality of each external sensing sensor. The observation quality, the inverse matrix of the measurement covariance matrix, and the three-dimensional position measurement vector are fused to calculate the threat three-dimensional coordinates of the target object. ; S12. Obtain the current attitude information of the operating platform, based on the threat 3D coordinates of the target object. Based on its own posture information, a relative coordinate system with the operator as the origin is established, and the relative pose of the target object relative to the operator is calculated. Within the relative coordinate system, the horizontal 360° azimuth plane centered on the operator is denoted as the overall azimuth region. The overall azimuth region is divided into eight directions according to the angle: front, back, left, right, left front, right front, left back, and right back. The eight directions are encoded separately, and an azimuth alarm constraint set D∈{1,2,3,…,8} is generated based on the relative pose. The numerical values ​​of each element in the azimuth alarm constraint set D are assigned to match the corresponding azimuth. Calculate the threat 3D coordinates of the target object The method is as follows:

[0028] in, For the first The threat's three-dimensional coordinates obtained from the secondary fusion For the first During the second fusion Normalized fusion weights assigned to observation data from external sensing sensors. For the first The external sensing sensor in the first The three-dimensional position measurement vector after coordinate unification during the second fusion. To and The inverse of the corresponding measurement covariance matrix, For the first The external sensing sensor in the first Observation quality during secondary fusion For the first The reliability coefficient of an external sensing sensor. This represents the total number of external sensing sensors.

[0029] The above scheme provides a specific method for generating the orientation alarm constraint set D. First, since each external sensing sensor outputs observation data of a target object, the observation data of each external sensing sensor is converted into a three-dimensional position measurement vector in the world coordinate system. Then, based on the observation uncertainty of each external sensing sensor, the inverse matrix of the measurement covariance matrix of each external sensing sensor is constructed. Specifically, the measurement covariance matrix (with diagonal elements representing the observation variance of the corresponding dimension) representing the observation uncertainty can be determined first based on the observation errors (such as ranging and angle measurement errors) of each external sensing sensor in each dimension (X / Y / Z axis) of three-dimensional space. Then, the inverse matrix of the measurement covariance matrix is ​​obtained through matrix inversion. Finally, the observation quality of each external sensing sensor is obtained, and the observation quality, the inverse matrix of the measurement covariance matrix, and the three-dimensional position measurement vector are fused to calculate the threat three-dimensional coordinates of the target object. The observation quality is quantified based on data from each sensor, such as the signal-to-noise ratio output by each sensor in real time; the specific target object's threat three-dimensional coordinates... It can be obtained through the following formula;

[0030] in, For the first The threat's three-dimensional coordinates obtained from the secondary fusion For the first During the second fusion Normalized fusion weights assigned to observation data from external sensing sensors. For the first The external sensing sensor in the first The three-dimensional position measurement vector after coordinate unification during the second fusion. To and The inverse of the corresponding measurement covariance matrix, For the first The external sensing sensor in the first Observation quality during secondary fusion For the first The reliability coefficient of an external sensing sensor can be determined based on its inherent performance parameters (such as the manufacturer's rated accuracy). This represents the total number of external sensing sensors. After obtaining the three-dimensional coordinates of the threat of the target object Then, the current attitude information of the operating platform (such as heading angle, pitch angle, roll angle, operator seating reference, etc.) is obtained, and a relative coordinate system is established with the operator as the origin. That is, with the operator's front as 0° as the reference, the horizontal 360° plane is the azimuth area, and the operator's position is the coordinate origin, to calculate the relative pose of the target object relative to the operator. Within the relative coordinate system, the horizontal 360° azimuth plane centered on the operator is recorded as the overall azimuth area. The overall azimuth area is divided into eight directions according to the angle: front, back, left, right, left front, right front, left back, and right back. The eight directions are encoded separately, and azimuth alarms are generated based on the relative pose. The bundle D∈{1,2,3,…,8} is used to assign numerical values ​​of each element in the azimuth alarm constraint set D to match the corresponding azimuth, thus providing accurate azimuth constraints for subsequent alarm modality decision-making. By introducing a weighted fusion algorithm that combines sensor type reliability and real-time observation quality, the observation error of a single sensor and the influence of environmental interference are effectively reduced, improving the accuracy of threat 3D coordinate calculation and ensuring the accuracy of azimuth identification from the root. At the same time, a relative coordinate system is established with the operator as the origin, and combined with equal angle division and standardized digital coding, a concrete and standardized representation of threat azimuth is realized, solving the problem of ambiguous azimuth definition and inability to accurately match alarm modality in existing technologies.

[0031] The method for establishing the task resource consumption feature vector T is as follows: the type information includes task type, operation frequency, interface interaction quantity, and control input vector. Based on the task type, operation frequency, interface interaction quantity, and control input vector, the interaction frequency is calculated within any time window Δt. Hearing occupancy ratio and control input intensity The data was then normalized to quantify the visual channel occupancy index. Auditory channel occupancy index and control channel occupancy indicators Map the occupancy indicators of each channel to a unified interval [0,1] to form a task resource occupancy feature vector. ; Calculate interaction frequency Hearing occupancy ratio and control input intensity The method is as follows: ; in, At the current assessment point, This represents the number of interface interactions within the time window. The cumulative duration of auditory output within the time window. For a moment Continuous control input vector; Visual channel occupancy index Auditory channel occupancy index and control channel occupancy indicators The quantification method is as follows:

[0032] in, , , These represent the task baseline occupancy rates for the visual, auditory, and control channels, respectively. , , These are the weighting coefficients for the visual channel, auditory channel, and control channel, respectively. as well as Interaction frequency The extreme values, as well as For auditory occupancy ratio The extreme values, as well as To control input intensity The extreme values, It is a preset minimum positive number.

[0033] The above scheme provides a specific method for establishing the task resource consumption feature vector T. First, the type information includes task type, operation frequency, interface interaction quantity, and control input vector. Based on the task type, operation frequency, interface interaction quantity, and control input vector, the interaction frequency is calculated within any time window Δt. Hearing occupancy ratio and control input intensity Specific computational interaction frequency Hearing occupancy ratio and control input intensity The method is as follows:

[0034] in, At the current assessment point, This represents the number of interface interactions within the time window. The cumulative duration of auditory output within the time window. For a moment Continuous control input vector; Then, it is normalized to obtain the visual channel occupancy index. Auditory channel occupancy index and control channel occupancy indicators The quantification formula is as follows:

[0035] in, , , These represent the baseline occupancy rates for the visual, auditory, and control channels, respectively, which can be obtained through historical task statistics or expert calibration. , , The weighting coefficients for the visual, auditory, and control channels are respectively, and can be determined based on experience and professional knowledge. as well as Interaction frequency The extreme values, as well as For auditory occupancy ratio The extreme values, as well as To control input intensity The extreme values, The preset minimum positive number is used to avoid the denominator being zero. The quantified occupancy indicators of each channel are mapped to a unified interval [0,1] and integrated in the order of "visual-auditory-control". Finally, a task resource occupancy feature vector T is generated. This vector can accurately quantify the occupancy status of the current control task on the operator's visual, auditory and control perception channels, and provide a standardized task constraint basis for subsequent alarm modal decision-making.

[0036] The method for constructing the situation awareness vector C is as follows: based on the set of physiological features Output attention level index The calculation model is as follows:

[0037] in, This represents the weight vector of the attention model. For the bias term of the attention model, This is the dimensionless eigenvector after baseline correction. For the purpose of using a set of physiological characteristics The feature vector extracted and concatenated within the current time window Feature vector of the operator in the baseline state The mean vector in each dimension, Feature vector of the operator in the baseline state Standard deviation vectors in each dimension; Based on physiological feature set Output cognitive load level indicators The calculation model is as follows:

[0038] in, This is the weight vector of the load model. This is the bias term for the load model. This is a truncation function; Cognitive load level index and attention level indicators By fusing the data, a situational awareness vector is obtained. .

[0039] The above scheme provides a specific method for constructing the situation awareness vector C, firstly based on a set of physiological features. Output attention level index The calculation model is as follows:

[0040] in, This represents the weight vector of the attention model. For the bias term of the attention model, This is the dimensionless eigenvector after baseline correction. For a set of physiological features The feature vector extracted and concatenated within the current time window Feature vector of the operator in the baseline state The mean vector in each dimension, Feature vector of the operator in the baseline state In the standard deviation vector of each dimension, The value is a preset minimum positive number; the attention level index is a core parameter obtained by quantifying physiological characteristics. It accurately represents the operator's current level of concentration and investment in the task, environment, or key information: the higher the value (closer to 1), the more concentrated the attention and the stronger the ability to perceive, process, and respond to information; the lower the value (closer to 0), the more scattered the attention, the easier it is to miss key information, and the significantly lower the efficiency and accuracy of response to stimuli such as alarms. Also based on a set of physiological characteristics Output cognitive load level indicators The calculation model is as follows:

[0041] in, This is the weight vector of the load model. This is the bias term for the load model. The truncation function is used; the cognitive load level index is a core parameter quantified based on standardized physiological characteristics. It accurately represents the degree of cognitive burden that the operator's brain bears in information processing and decision-making under the current task execution and environmental stimuli. The larger the value of this index, the higher the proportion of the operator's cognitive resources occupied by the current task, and the heavier the brain's information processing load. When the value approaches 1, it indicates that the operator has reached a state of cognitive saturation and can no longer effectively receive or process additional information stimuli. Therefore, the cognitive load level index and attention level indicators By fusing the data, a situational awareness vector is obtained. Situational cognition vector It can characterize the operator's current overall cognitive state and cognitive resource suitability. By combining the two, the level of cognitive risk can be accurately determined: for example, when Approaching 1 (cognitive saturation) and When the value approaches 0 (indicating distraction), it indicates that the operator is in a state of high cognitive risk, and additional information input should be avoided at this time; when... lower and A higher value indicates that the operator's cognitive state is good, and they can effectively receive and process alarm information. By integrating attention and cognitive load into a unified situational awareness vector C in this way, the optimal decision-making model for subsequent multimodal alarms is perfectly adapted. This provides core data support for intelligent adaptation of alarm modes based on cognitive state, effectively avoids invalid alarms when cognitive overload occurs and information omissions when attention is distracted, and significantly improves the safety and scientific nature of human-computer interaction systems.

[0042] The method for calculating the applicability evaluation value is as follows: the candidate alarm mode set is... ,in Indicates visual alarm modality, Indicates auditory alarm mode, Representing tactile alarm modality; [This refers to] the task resource usage feature vector. Situational cognition vector The evaluation input vector X is obtained by normalizing and combining the azimuth alarm constraint set D; based on the evaluation input vector... Establish an optimization model with the objective of maximizing the applicability evaluation value of directional information transmission; based on the optimization model, analyze the candidate alarm mode set. The applicability evaluation value for each alarm mode is calculated and output. The applicability evaluation value model is as follows:

[0043] in, For alarm mode The applicability evaluation value of location information transmission under given conditions. , , Alarm modes The incremental resource usage of the visual, auditory, and control channels. For alarm mode Reliability at azimuth D; The method for outputting the optimal alarm mode is as follows: The optimization model is as follows:

[0044] in, This indicates whether to select the alarm mode. Decision variables, , , These are the safety margins for the visual, auditory, and control channels, respectively. For alarm mode The feasibility indicator is determined by the optimization model, which yields the evaluation values ​​of each alarm mode. The alarm mode with the highest evaluation value is then selected as the optimal alarm mode for output.

[0045] The above scheme provides a final determination of the optimal alarm mode and triggers the corresponding mode output. First, the candidate alarm mode set is set as follows: ,in Indicates visual alarm modality, Indicates auditory alarm mode, Representing tactile alarm modality; [This refers to] the task resource usage feature vector. Situational cognition vector The evaluation input vector X is obtained by normalizing and combining the azimuth alarm constraint set D. Based on the evaluation input vector Establish an optimization model with the objective of maximizing the applicability evaluation value of directional information transmission; based on the optimization model, analyze the candidate alarm mode set. The algorithm solves for each alarm mode and outputs the applicability evaluation value for each alarm mode. The applicability evaluation value model is as follows:

[0046] in, For alarm mode The applicability evaluation value of location information transmission under given conditions. , , Alarm modes The incremental resource usage of the visual, auditory, and control channels is defined as the channel usage after an alarm occurs. Channel occupancy before the alarm occurred The difference: , For alarm mode The reliability of azimuth D can be obtained by statistically analyzing the accuracy of the alarm mode in identifying that azimuth in pilot experiments; the applicability evaluation value indicates the effectiveness of the alarm mode under current task resource constraints, operator situational awareness, and target azimuth constraints. The overall effectiveness and adaptability of transmitting location information are evaluated. The closer the value is to 1, the better the modality is at transmitting location information in the current scenario, and the higher its adaptability. Based on the applicability evaluation value and the decision variables of each alarm modality, the optimal alarm modality is output and the alarm information is triggered. The optimization model is as follows:

[0047] in This indicates whether to select the alarm mode. Decision variables, For alarm mode Feasibility indicator, when the orientation For alarm modes Expression is limited, or when attention level indicators Low cognitive load level indicators Excessive levels make it unsuitable to use alarm modes. hour, This prevents the mode from being selected; otherwise, it takes the value 1. , , These represent safety margins for the visual, auditory, and control channels, respectively, to prevent channel saturation. Their initial values ​​are... The settings can be obtained by referring to experimental data and experience. The general adjustment method for each safety margin is as follows:

[0048] in, For channel System latency, For channel Maximum delay, For channel The initial value of the safety margin. For channel Evaluation window Indicates the visual channel. Indicates the auditory channel. This represents the control channel; based on the solution results of the optimization model, let satisfy... The alarm mode is the optimal alarm mode, and the corresponding alarm mode outputs directional alarm information. The optimization model constructed in this solution integrates task resource consumption constraints, cognitive feasibility constraints, and safety margins, ensuring the effective transmission of alarm information while avoiding cognitive interference of new alarms on operators' existing tasks, achieving a dual balance between maximizing adaptability and minimizing resource consumption. Through decision variables... Feasibility indicators The linkage enables precise elimination of incompatible modalities, significantly improving decision-making efficiency and accuracy; and provides a standardized and implementable decision core for multimodal directional alarm systems, effectively improving the transmission efficiency and security of alarm information in complex human-computer interaction scenarios, and reducing security risks caused by incompatible alarm modalities.

[0049] One embodiment discloses a multimodal azimuth alarm configuration system based on situational awareness, such as Figure 3 As shown, the system includes: The orientation alarm constraint set generation module is used to deploy an external sensing sensor group on the operating platform to scan the surrounding environment of the target object in real time, acquire multi-source sensor data, analyze and calculate the multi-source sensor data, obtain the threat three-dimensional coordinates of the target object, and generate the orientation alarm constraint set D based on the threat three-dimensional coordinates. The task resource acquisition and evaluation module is used to acquire the type information of the current operation task, calculate the channel occupancy index of the current operation task based on the type information, and establish a task resource occupancy feature vector T based on each channel occupancy index. The physiological feature extraction module is used to acquire the operator's physiological signals, preprocess the physiological signals and extract features to obtain physiological features, and construct a physiological feature set P based on the physiological features; The situational awareness assessment module is used to calculate attention level indicators and cognitive load level indicators based on the physiological feature set P, and to construct a situational awareness vector C based on the attention level indicators and cognitive load level indicators. The multimodal alarm decision module is used to determine the candidate alarm mode set M. With the task resource occupancy feature vector T, situational awareness vector C, and azimuth alarm constraint set D as constraints, it calculates the applicability evaluation value of each alarm mode in the candidate alarm mode set M for the transmission of azimuth information. Based on the applicability evaluation value and the decision variables of each alarm mode, it outputs the optimal alarm mode and triggers alarm information.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0051] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0052] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal azimuth alarm configuration method based on situational awareness, characterized in that, The method includes: S1. Deploy an external sensing sensor group on the operating platform to scan the surrounding environment of the target object in real time, acquire multi-source sensor data, analyze and calculate the multi-source sensor data, obtain the threat three-dimensional coordinates of the target object, and generate a directional alarm constraint set D based on the threat three-dimensional coordinates. S2. Obtain the type information of the current operation task, calculate the channel occupancy index of the current operation task based on the type information, and establish the task resource occupancy feature vector T based on each channel occupancy index; S3. Acquire the physiological signals of the operator, preprocess the physiological signals and extract features to obtain physiological features, and construct a set of physiological features P based on the physiological features; S4. Calculate the attention level index and cognitive load level index based on the physiological feature set P, and construct the situational cognition vector C based on the attention level index and cognitive load level index. S5. Determine the candidate alarm mode set M. Using the task resource occupancy feature vector T, situational awareness vector C, and azimuth alarm constraint set D as constraints, calculate the applicability evaluation value of each alarm mode in the candidate alarm mode set M for azimuth information transmission. Based on the applicability evaluation value and the decision variables of each alarm mode, output the optimal alarm mode and trigger the alarm information.

2. The multimodal azimuth alarm configuration method based on situational awareness according to claim 1, characterized in that, The steps for generating the azimuth alarm constraint set D are as follows: S11. Multi-source sensor data includes observation data and the current attitude information of the operating platform. Each external sensing sensor outputs observation data of a target object. The observation data of each external sensing sensor is converted into a three-dimensional position measurement vector in the world coordinate system. Based on the observation uncertainty of each external sensing sensor, the inverse matrix of the measurement covariance matrix of each external sensing sensor is constructed to obtain the observation quality of each external sensing sensor. The observation quality, the inverse matrix of the measurement covariance matrix, and the three-dimensional position measurement vector are fused to calculate the threat three-dimensional coordinates of the target object. ; S12. Obtain the current attitude information of the operating platform, based on the threat 3D coordinates of the target object. Based on its own posture information, a relative coordinate system with the operator as the origin is established, and the relative pose of the target object relative to the operator is calculated. Within the relative coordinate system, the horizontal 360° azimuth plane centered on the operator is denoted as the overall azimuth region. The overall azimuth region is divided equally into eight directions according to angle: front, back, left, right, left front, right front, left back, and right back. The eight directions are encoded separately, and an azimuth alarm constraint set D∈{1,2,3,…,8} is generated based on the relative pose. The numerical values ​​of each element in the azimuth alarm constraint set D are matched with the corresponding azimuth.

3. The multimodal azimuth alarm configuration method based on situational awareness according to claim 2, characterized in that, Calculate the threat 3D coordinates of the target object The method is as follows: in, For the first The threat's three-dimensional coordinates obtained from the secondary fusion For the first During the second fusion Normalized fusion weights assigned to observation data from external sensing sensors. For the first The external sensing sensor in the first The three-dimensional position measurement vector after coordinate unification during the second fusion. To and The inverse of the corresponding measurement covariance matrix, For the first The external sensing sensor in the first Observation quality during secondary fusion For the first The reliability coefficient of an external sensing sensor. This represents the total number of external sensing sensors.

4. The multimodal azimuth alarm configuration method based on situational awareness according to claim 1, characterized in that, The method for establishing the task resource consumption feature vector T is as follows: Type information includes task type, operation frequency, interface interaction quantity, and control input vector. Based on task type, operation frequency, interface interaction quantity, and control input vector, the interaction frequency is calculated within any time window Δt. Hearing occupancy ratio and control input intensity The data was then normalized to quantify the visual channel occupancy index. Auditory channel occupancy index and control channel occupancy indicators Map the occupancy indicators of each channel to a unified interval [0,1] to form a task resource occupancy feature vector. .

5. The multimodal azimuth alarm configuration method based on situational awareness according to claim 4, characterized in that, Calculate interaction frequency Hearing occupancy ratio and control input intensity The method is as follows: ； in, At the current assessment point, This represents the number of interface interactions within the time window. The cumulative duration of auditory output within the time window. For a moment Continuous control input vector; Visual channel occupancy index Auditory channel occupancy index and control channel occupancy indicators The quantification method is as follows: in, , , These represent the task baseline occupancy rates for the visual, auditory, and control channels, respectively. , , These are the weighting coefficients for the visual channel, auditory channel, and control channel, respectively. as well as Interaction frequency The extreme values, as well as For auditory occupancy ratio The extreme values, as well as To control input intensity The extreme values, It is a preset minimum positive number.

6. The multimodal azimuth alarm configuration method based on situational awareness according to claim 1, characterized in that, The method for constructing the situation awareness vector C is as follows: Based on physiological feature set Output attention level index The calculation model is as follows: in, This represents the weight vector of the attention model. For the bias term of the attention model, This is the dimensionless eigenvector after baseline correction. For a set of physiological features The feature vector extracted and concatenated within the current time window Feature vector of the operator in the baseline state The mean vector in each dimension, Feature vector of the operator in the baseline state In the standard deviation vector of each dimension, It is a preset minimum positive number; Based on physiological feature set Output cognitive load level indicators The calculation model is as follows: in, This is the weight vector of the load model. This is the bias term for the load model. This is a truncation function; Cognitive load level index and attention level indicators By fusing the data, a situational awareness vector is obtained. .

7. The multimodal azimuth alarm configuration method based on situational awareness according to claim 1, characterized in that, The method for calculating the applicability evaluation value is as follows: The candidate alarm modal set is ,in Indicates visual alarm modality, Indicates auditory alarm mode, Indicates tactile alarm mode; Task resource consumption feature vector Situational cognition vector The evaluation input vector X is obtained by normalizing and combining the azimuth alarm constraint set D; based on the evaluation input vector... Establish an optimization model with the objective of maximizing the applicability evaluation value of directional information transmission; based on the optimization model, analyze the candidate alarm mode set. The applicability evaluation value for each alarm mode is calculated and output. The applicability evaluation value model is as follows: in, For alarm mode The applicability evaluation value of location information transmission under given conditions. , , Alarm modes The incremental resource usage of the visual, auditory, and control channels. For alarm mode Reliability in orientation D.

8. The multimodal azimuth alarm configuration method based on situational awareness according to claim 7, characterized in that, The method for outputting the optimal alarm mode is as follows: The optimal model is: in, This indicates whether to select the alarm mode. Decision variables, , , These are the safety margins for the visual, auditory, and control channels, respectively. For alarm mode Feasibility indicators; The evaluation values ​​of each alarm mode are obtained through the optimization model, and the alarm mode with the largest evaluation value is selected as the optimal alarm mode for output.

9. A multimodal azimuth alarm configuration method based on situational awareness according to claim 8, characterized in that, Safety margins for visual, auditory, and control channels , , The method for obtaining it is as follows: The general method for adjusting the safety margin is as follows: in, For channel System latency, For channel Maximum delay, For channel The initial value of the safety margin. For channel Evaluation window Indicates the visual channel. Indicates the auditory channel. Indicates the control channel.

10. A multimodal azimuth alarm configuration system based on situational awareness, implementing the multimodal azimuth alarm configuration method based on situational awareness as described in any one of claims 1-9, characterized in that, The system includes: The orientation alarm constraint set generation module is used to deploy an external sensing sensor group on the operating platform to scan the surrounding environment of the target object in real time, acquire multi-source sensor data, analyze and calculate the multi-source sensor data, obtain the threat three-dimensional coordinates of the target object, and generate the orientation alarm constraint set D based on the threat three-dimensional coordinates. The task resource acquisition and evaluation module is used to acquire the type information of the current operation task, calculate the channel occupancy index of the current operation task based on the type information, and establish a task resource occupancy feature vector T based on each channel occupancy index. The physiological feature extraction module is used to acquire the operator's physiological signals, preprocess the physiological signals and extract features to obtain physiological features, and construct a physiological feature set P based on the physiological features; The situational awareness assessment module is used to calculate attention level indicators and cognitive load level indicators based on the physiological feature set P, and to construct a situational awareness vector C based on the attention level indicators and cognitive load level indicators. The multimodal alarm decision module is used to determine the candidate alarm mode set M. With the task resource occupancy feature vector T, situational awareness vector C, and azimuth alarm constraint set D as constraints, it calculates the applicability evaluation value of each alarm mode in the candidate alarm mode set M for the transmission of azimuth information. Based on the applicability evaluation value and the decision variables of each alarm mode, it outputs the optimal alarm mode and triggers alarm information.

Images