A High-Dimensional Adaptive Display and Control Method for Sonar Images Based on Cognitive State Closed-Loop Regulation

By combining sonar front-end processing and operator multimodal data, the cognitive state is estimated in real time, and individualized display control actions are generated. This solves the problems of information overload and low interaction efficiency in sonar image display, and achieves the effect of prioritizing the presentation of key information and low-latency response.

CN123067764APending Publication Date: 2026-11-10ZHEJIANG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611336641.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-31
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing sonar image display technology lacks closed-loop control over the operator's cognitive state, resulting in information overload, low interaction efficiency, insufficient evaluation of target saliency, and the inability to dynamically allocate edge computing resources, making it difficult to adapt to different tasks and operator differences.

Method used

By acquiring high-dimensional display elements from the sonar front-end processing module, and combining them with task context and operator multimodal data, cognitive load and fatigue status are estimated in real time, display control actions are generated, and a closed-loop display strategy is formed to adapt to different task stages and individualized needs of operators.

Benefits of technology

It reduces the overload of high-dimensional display information, improves interactive security and efficiency, adapts to the display needs of different platforms and operators, ensures that key information is presented first, reduces the risk of misoperation, and achieves low-latency response at the edge.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123067764A_ABST
    Figure CN123067764A_ABST
Patent Text Reader

Abstract

This invention relates to the field of sonar image display technology, specifically to a high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state. The method first abstracts the high-dimensional display elements output by the sonar front-end processing module into unified display objects; constructs a task context and calculates the acoustic saliency and display priority of each display object; estimates the operator's cognitive load, fatigue state, interaction intent, and intent confidence in real time based on the operator's multimodal data; inputs the display priority, task context, cognitive load, fatigue state, interaction intent, and intent confidence into a closed-loop display strategy engine to generate display control actions; executes the display control actions to update the sonar high-dimensional display interface; and performs lightweight processing on the model at the edge, updating individualized parameters based on the operator's historical interaction data. This invention enables the same sonar display and control system to adapt to different personnel, different task stages, and different platform display conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of sonar image display technology, and more specifically to a high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation. Background Technology

[0002] High-dimensional sonar image display technology comprehensively expresses underwater acoustic signals in multiple dimensions, including time, space, frequency, sound pressure, and target confidence, providing intuitive information for underwater target detection, identification, obstacle avoidance, and mission decision-making. With the development of active sonar, passive sonar, and unmanned underwater platforms, the amount of information that sonar display and control interfaces can present is rapidly increasing. The displayed objects have also expanded from a single two-dimensional distance-azimuth map to multiple elements such as time-frequency maps, three-dimensional scenes, historical trajectories, confidence layers, and alarm information.

[0003] Existing technologies have made considerable progress in improving the resolution and visualization dimensions of sonar images, but most solutions still primarily aim to "display more information," lacking real-time awareness of the operator's current task, attention distribution, cognitive load, and fatigue state. In complex sea conditions, strong reverberation, long-term monitoring, or multi-target scenarios, the simultaneous appearance of numerous layers, candidate targets, and time-series information on the interface can easily lead to information congestion, longer visual search times, and key targets being obscured by non-critical noise.

[0004] Some sonar systems have begun to introduce multimodal interaction methods such as voice, gestures, and eye tracking. However, these interaction methods are usually only used as input channels and have not yet formed a closed-loop linkage with the high-dimensional display strategy of sonar images. For example, the system can recognize the operator's instructions to switch perspectives or mark targets, but it cannot actively adjust the layer density, time window, alarm level, and target highlighting method based on the operator's gaze duration, scan rate, instruction lag, frequency of misoperations, etc.

[0005] Furthermore, edge platforms such as shipborne terminals, buoy nodes, and unmanned underwater vehicles are limited in terms of computing resources, display area, and communication capabilities. Simply increasing rendering accuracy or running multiple complex interactive models simultaneously can easily lead to display latency and delayed command response; conversely, using fixed downsampling or a fixed interface layout makes it difficult to adapt to different mission phases and varying cognitive abilities of different operators.

[0006] In summary, existing technologies have at least the following shortcomings: First, there is a lack of closed-loop regulation between high-dimensional sonar display information and the operator's cognitive state, which can easily lead to information overload; second, multimodal interaction data is not fully utilized for adaptive display strategies, and interaction efficiency and security remain limited; third, there is a lack of a unified saliency evaluation mechanism among targets, noise, trajectories, confidence levels, and task priorities; fourth, edge computing resources cannot be dynamically allocated based on task urgency and cognitive load; and fifth, differences in operating habits and fatigue thresholds among different operators are not effectively utilized, resulting in insufficient individualized adaptability of the system. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention aims to provide a high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive states. The specific technical solution adopted is as follows: Acquire high-dimensional display elements output by the sonar front-end processing module, and abstract these high-dimensional display elements into a unified display object; The task context is constructed by combining the current task stage and the operator's operation goal, and the acoustic saliency and display priority of each displayed object are calculated; Collect multimodal data of the operator, and estimate the operator's cognitive load, fatigue state, interaction intention and intention confidence in real time based on the multimodal data; The display priority, task context, cognitive load, fatigue state, interaction intent, and intent confidence are input into the closed-loop display strategy engine to generate display control actions; The display control action is executed to update the sonar high-dimensional display interface and provide fatigue warning; The cognitive state estimation model and the display strategy model are lightweighted at the edge, and the individualized parameters are updated based on the operator's historical interaction data.

[0008] The embodiments of the present invention have at least the following beneficial effects: First, this invention reduces the overload of high-dimensional display information. Instead of simply increasing the display dimension of sonar images, it determines the displayed content through acoustic saliency, task context, and operator cognitive state. This prioritizes the presentation of key targets, hazardous areas, and high-confidence information, while weakening or aggregating low-priority noise and reverberation, thus reducing the burden of manual interpretation. Second, it forms a cognitive state-driven closed-loop display and control system. The system estimates the operator's cognitive load, fatigue state, interaction intent, and intent confidence in real time and feeds this information back to the display strategy engine. This enables adaptive adjustments to layer density, viewpoint focus, time window, and alarm level, better adapting to long-term, highly disruptive, and multi-target operation scenarios compared to a fixed interface layout. Third, it improves interaction security. Through multimodal intent fusion and secondary confirmation of key commands, this invention can quickly execute commands when voice, gestures, eye movements, or operation logs corroborate each other. It automatically requests confirmation when there is modal conflict or increased fatigue, reducing operational risks such as false locking, false alarm cancellation, and false switching. Fourth, it is suitable for real-time deployment at the edge. This invention deploys the cognitive state estimation model, display strategy model, and task lookup table at the edge, allowing continued operation even when communication is unstable or the central server is unavailable. Through lightweight model design and adaptive resource allocation, it ensures low-latency response in critical display and control links. Fifth, it possesses individualized adaptability. The system can perform local parameter calibration based on different operators' gaze habits, command speeds, and fatigue thresholds, enabling the same sonar display and control system to adapt to different personnel, different task stages, and different platform display conditions, thus improving the flexibility of engineering applications. Attached Figure Description

[0009] Figure 1 A flowchart of a high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation is provided in one embodiment of the present invention. Figure 2 A schematic diagram of the system structure for high-dimensional display of cognitive closed-loop sonar provided in an embodiment of the present invention; Figure 3 Another flowchart of a high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation is provided as an embodiment of the present invention. Detailed Implementation

[0010] The following description, in conjunction with the accompanying drawings, details a specific scheme for a high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation provided by the present invention.

[0011] Please see Figure 1 The diagram illustrates a flowchart of a high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation, according to an embodiment of the present invention. The method includes the following steps: Step S100: Obtain the high-dimensional display elements output by the sonar front-end processing module, and abstract the high-dimensional display elements into a unified display object.

[0012] Step S110: Obtain the high-dimensional display elements output by the sonar front-end processing module. Specifically, receive the range-azimuth map, time-frequency map, 3D point cloud or volume data, target candidate list, historical trajectory, noise / reverberation estimation results, and confidence score of each target candidate output by the sonar front-end processing module. The above data can come from traditional beamforming, deep learning recognition, matched filtering, tracking filtering, or other sonar processing algorithms; this invention does not limit its specific source.

[0013] As one embodiment of the present invention, the sonar front-end processing module includes a sonar front-end processing algorithm, which can be a traditional beamforming, matched filtering, deep learning recognition, or other target detection algorithm.

[0014] Step S120: The high-dimensional display elements are uniformly abstracted into display objects. Specifically, the accessed data is uniformly abstracted into display objects oi. The display object oi includes at least object type, spatial location, timestamp, spectral characteristics, echo intensity, confidence level, motion state, occlusion relationship, and visualization layer identifier. Object types include target echo, suspected target, background noise, reverberation area, terrain / obstacle, historical trajectory, and alarm marker.

[0015] As another embodiment of the present invention, the display object may consist of any one or more of the following: two-dimensional sonar image, three-dimensional point cloud, volume data, target list, or historical track.

[0016] Finally, time synchronization and caching are performed. Various display objects are synchronized according to a unified timeline to form a sliding time window of length W, and the object states of the most recent N windows are cached at the edge for subsequent saliency calculation, historical playback, and individualized parameter updates.

[0017] Step S200: Construct a task context by combining the current task stage and the operator's operation goal, and calculate the acoustic saliency and display priority of each displayed object.

[0018] Step S210: Construct a task context by combining the current task stage and the operator's operational objectives. Specifically, based on the operator's selection, platform navigation status, and task plan, the current task stage is divided into stages such as search, tracking, obstacle avoidance, verification, historical playback, and alarm handling, and a task context vector M(t) is generated. The task context vector includes parameters such as task type, target priority, platform speed, hazardous area, remaining handling time, and available display area.

[0019] As another embodiment of the present invention, the task context may also be provided automatically by manual selection, a task planning system, or a platform control system.

[0020] Step S220: Calculate the acoustic saliency and display priority of each display object, specifically: Obtain the echo intensity, spectral concentration, trajectory anomaly, confidence level, task relevance, and occlusion penalty of the displayed object; Taking the i-th display object oi as an example, the acoustic saliency Ai is calculated. The acoustic saliency is jointly determined by the echo intensity qi, spectral concentration ri, trajectory anomaly di, confidence ui, task relevance hi, and occlusion penalty bi. Ai=w1×qi+w2×ri+w3×di+w4×ui+w5×hi-w6×bi (1) (2) Where pi is the normalized display priority of the i-th display object, m is the total number of display objects in the current time window, Aj represents the acoustic saliency of the j-th display object, and τ is the temperature coefficient. When τ is small, the system tends to highlight a few highly saliency objects; when τ is large, the priority distribution of each object is smoother, which is suitable for wide-field observation during the search phase.

[0021] Among them, w1, w2, w3, w4, w5, and w6 are weight parameters that can be dynamically adjusted according to the task context. When the task is in the search phase, the weights of echo intensity and spectral concentration are increased; when the task is in the tracking phase, the weights of trajectory anomaly and target confidence are increased; when the task is in the obstacle avoidance phase, the weight w5 corresponding to the task relevance hi is increased. Among them, the task relevance hi is used to characterize the degree of association between the displayed object and the current task phase, and the value range is [0,1]. In the obstacle avoidance phase, hi is determined by the spatial relationship between the displayed object and the danger zone and the platform: when the displayed object is located in the danger zone marked by the task context M(t), or when its relative distance li to the platform is less than the distance threshold dth, hi=1; when li≥dth, hi decreases linearly with the increase of distance, i.e., hi=max(0,1-(li-dth) / dth). The distance threshold dth is determined by the platform's current speed v and the reserved obstacle avoidance reaction time Tr, i.e., dth = v × Tr. As an embodiment of the present invention, Tr is taken as 30 s to 60 s. When the platform speed is 5 m / s and Tr is taken as 40 s, dth is 200 m. The danger zone is given by the danger zone field in the mission context M(t), which can be derived from nautical chart depth data, terrain / obstacle detection results, or manually marked no-navigation zones.

[0022] More specifically, the dynamic adjustment of the weight parameters is performed according to a pre-established "task stage - weight template" lookup table: using the task type field in the task context M(t) as an index, the weight vector w=[w1,w2,w3,w4,w5,w6] corresponding to the task stage is retrieved, and the value range of each weight is [0,1] and satisfies w1+w2+w3+w4+w5+w6=1. As a specific embodiment of the present invention, w=[0.30,0.25,0.15,0.15,0.10,0.05] can be used in the search stage, w=[0.15,0.10,0.25,0.25,0.20,0.05] can be used in the tracking stage, and w=[0.15,0.10,0.15,0.15,0.40,0.05] can be used in the obstacle avoidance stage. When a task phase changes, the weights undergo a first-order smooth transition according to w(t) = μ·w(t-1) + (1-μ)·w_target, where μ is the smoothing coefficient, ranging from 0.5 to 0.9, to avoid abrupt changes in saliency ranking and interface layout. The initial values ​​of the above weight template can be obtained through offline calibration and individually calibrated based on the operator's historical interaction data.

[0023] As another embodiment of the present invention, acoustic saliency calculation can also be achieved using linear weighting, hierarchical analysis, Bayesian scoring, learning ranking models, or reinforcement learning value functions.

[0024] Then, objects are sorted according to display priority. High-priority objects are given higher opacity, more obvious borders, higher refresh rates, or more prominent layer positions in the display interface; low-priority objects are made semi-transparent, clustered, or temporarily hidden.

[0025] Step S300: Collect multimodal data of the operator, and estimate the operator's cognitive load, fatigue state, interaction intention and intention confidence in real time based on the multimodal data.

[0026] Step S310: Collect multimodal data of the operator. The multimodal data includes voice commands, gestures, gaze points, pupil changes, blink frequency, EEG rhythm characteristics, mouse / knob / touch operations, and command response time; Specifically, the system uses a microphone, gesture camera, eye tracker, EEG acquisition device, and display / control terminal log module to acquire the operator's voice commands, gestures, gaze points, pupil changes, blink frequency, EEG rhythm characteristics, mouse / knob / touch operations, and command response time. For scenarios where not all sensors are available, only one or more modalities can be collected.

[0027] As another embodiment of the present invention, the multimodal data can be a subset selected according to the device conditions, such as using only blink frequency and mouse / knob / touch operation logs, or using only voice commands and gestures.

[0028] Step S320, Feature Extraction and Alignment. Using the sonar display frame as a reference, the multimodal data is mapped to a unified time window, and features such as gaze duration, scan rate, pupil diameter change rate, blink interval, voice command confidence, gesture completion time, number of misoperations, and EEG band energy are extracted to obtain the operator state feature vector H(t).

[0029] Step S330: Based on multimodal data, estimate the operator's cognitive load, fatigue state, interaction intention, and intention confidence in real time. Specifically: Input the state feature vector H(t), task context M(t), and the most recent interaction actions into a lightweight temporal attention model or a gated recurrent model, and output the operator's cognitive load C(t), fatigue state F(t), interaction intention U(t), and intention confidence R(t): [C(t), F(t), U(t), R(t)] = fθ(H(t), M(t), A(t)) (3) C(t) = σ(αc) T H(t)+βc T M(t)+γc T Ā(t)+bc) (4) F(t) = ρF(t-1) + (1-ρ)σ(αf T H(t)+βf T E(t)+bf) (5) Where C(t) represents cognitive load, F(t) represents fatigue state, U(t) represents interaction intent, R(t) represents intent confidence, Ā(t) is the saliency statistic of the currently displayed object, E(t) is the fatigue feature vector composed of continuous working time, blink frequency, and number of misoperations, σ is the Sigmoid function, M(t) is the task context, A(t) is the acoustic saliency set of the currently displayed object, and fθ is the cognitive state estimation model deployed at the edge; the superscript ᵀ indicates vector transpose; αc, βc, and γc are the weight vectors corresponding to the state feature vector H(t), task context vector M(t), and saliency statistic Ā(t) in cognitive load estimation, respectively, and their dimensions are the same as those of H(t), M(t), and γc, respectively. = ), Ā(t) are the same, bc is the bias term of cognitive load estimation, which is a scalar; αf and βf are the weight vectors corresponding to the state feature vector H(t) and fatigue feature vector E(t) in fatigue state estimation, respectively, and their dimensions are the same as H(t) and E(t), respectively. bf is the bias term of fatigue state estimation, which is a scalar; the above weight vectors and bias terms together constitute the parameter set θ of the cognitive state estimation model fθ, that is, θ={αc,βc,γc,bc,αf,βf,bf}, whose values ​​are obtained by training with the operator's subjective load scale score and job performance index as labels in the offline training stage using the gradient descent method, and can be individually fine-tuned according to the operator's historical interaction data. This invention does not limit its specific value. ρ is the time memory coefficient of fatigue state, and the value range is 0<ρ<1. The larger ρ is, the stronger the historical cumulative effect of fatigue state and the less sensitive it is to instantaneous fluctuations. As an embodiment of this invention, ρ is taken as 0.9.

[0030] Where Ā(t) is the saliency statistic of the currently displayed object, which is a fixed-dimensional feature vector obtained by statistically aggregating the acoustic saliency set A(t)={A1,A2,…,Am} of all displayed objects within the current time window. As an embodiment of the present invention, Ā(t)=[mean(A(t)), max(A(t)), var(A(t)), m, m_high], where mean(A(t)) is the mean of the acoustic saliency set, max(A(t)) is the maximum value of the acoustic saliency set, var(A(t)) is the variance of the acoustic saliency set, m is the total number of displayed objects within the current time window, and m_high is the number of displayed objects whose acoustic saliency is higher than a set saliency threshold. Since the total number of displayed objects m changes dynamically with the scene, the dimension is not fixed when directly using the set A(t) as the model input. Therefore, the statistic Ā(t) is used as a fixed-length input to characterize the overall information content and congestion level of the current screen. The saliency threshold Ath is used to distinguish between salient and non-salient objects in the current screen. Since all feature components in formula (1) have been normalized to [0,1] and the sum of their weights is 1, the acoustic saliency Ai ranges from [-w6,1]. As an embodiment of the present invention, a fixed threshold Ath = 0.6 can be used, that is, display objects with acoustic saliency not less than 0.6 are included in m_high. As another embodiment of the present invention, Ath can also be set adaptively, that is, the mean of the acoustic saliency of all display objects in the current time window plus one standard deviation is taken, Ath = mean(A(t)) + √var(A(t)); or the 75th percentile of the acoustic saliency distribution in the current time window is taken. When the adaptive setting method is used, Ath is automatically adjusted with the scene target density, which can avoid m_high being 0 or equal to m in weak echo or strong reverberation scenes. The above threshold can also be individually calibrated according to the operator's historical interaction data.

[0031] The fatigue feature vector E(t) = [e1(t), e2(t), e3(t)]ᵀ, where e1(t) is the operator's continuous working time, e2(t) is the blink frequency, and e3(t) is the number of misoperations per unit time. Each component is normalized to [0,1] before being input into the model. As another embodiment of the present invention, the fatigue feature vector may also include components such as gaze dispersion and pupil diameter change rate, or only one or more of these components may be retained when the sensor is limited.

[0032] To avoid the model from relying too heavily on a single sensor, this invention sets a modality missing mask at the model input. When a certain modality is unavailable, the model automatically reduces the weight of that modality and continues inference using the remaining modalities.

[0033] As an embodiment of the present invention, the cognitive state estimation model can be a support vector machine, random forest, recurrent neural network, Transformer model or rule threshold model.

[0034] In step S400, the display priority, task context, cognitive load, fatigue state, interaction intent, and intent confidence are input into the closed-loop display strategy engine to generate display control actions.

[0035] Step S410: Generate a display strategy and constrain information load. Specifically, taking time t as an example, the closed-loop display strategy engine receives the display priority pi, task context M(t), cognitive load C(t), fatigue state F(t), interaction intent U(t), and intent confidence R(t), and outputs the display control action at. The display control action includes: layer on / off, object aggregation granularity, volume data sampling density, time window length, target annotation intensity, alarm sound level, viewpoint focus, color scheme, and command confirmation mode.

[0036] at=π(pi, M(t), C(t), F(t), U(t), R(t)) (6) (7) (8) Where, at is the display control action output at time t, π represents the display strategy function implemented by the closed-loop display strategy engine, that is, the mapping from display priority, task context and operator cognitive state to display control actions; i is the number index of the display object, i=1,2,…,m, m is the total number of display objects in the current time window; Ω is the set of display control actions that the display control terminal can execute at the current time, that is, the action set; a is any feasible display control action; π* represents the optimal display strategy that maximizes the objective function in formula (7); pi is the first... Normalized display priority of i display objects; Vi(a) represents the visibility gain of display control action a on the i-th display object, Ld(a) is the jump penalty between two adjacent interface actions, Lc(a) is the cognitive load penalty, Le(a) is the edge computing overhead penalty; Tkey(a) represents the critical instruction link latency, Tmax is the maximum allowed response time, C(t) represents the cognitive load; R(t) is the intent confidence; Bmax is the display load limit; bi(a) represents the information load of display control action a on the i-th display object.

[0037] Wherein, λ1, λ2, and λ3 are the weight coefficients of the jump penalty, cognitive load penalty, and edge computing overhead penalty, respectively, all of which are adjustable constants greater than 0, used to balance the visibility gains with the above three types of costs. As an embodiment of the present invention, the values ​​of λ1, λ2, and λ3 are all in the range of 0.1 to 1.0, with typical values ​​of λ1=0.3, λ2=0.5, and λ3=0.2. The larger λ1 is, the more stable the interface layout and the fewer the switching; the larger λ2 is, the more the strategy engine tends to simplify the displayed content to reduce the cognitive burden on the operator; the larger λ3 is, the more priority is given to saving edge computing power. The above coefficients can be dynamically tuned according to the operator's state, for example, when the fatigue state F(t) increases, λ1 and λ2 are increased, and when the remaining computing power of the edge node decreases, λ3 is increased.

[0038] As another embodiment of the present invention, the display control actions can be tailored according to the actual display and control terminal capabilities, retaining one or more of the following: layer switching, target highlighting, time window adjustment, and alarm classification.

[0039] Set a display load limit constraint. To prevent sudden interface congestion, the system sets a display load limit Bmax, which can be determined by a fixed threshold, manual selection by the operator, or a task template. Each display object (oi) corresponds to an information load. The strategy engine ensures that the current screen's display load meets the limit while prioritizing the display of critical objects. . When the cognitive load C(t) exceeds the first threshold, the system reduces low-priority noise layers, enlarges high-priority target annotations and shortens the alarm information text; when C(t) exceeds the second threshold, the system switches to a simplified display and control mode, retaining only targets, tracks, dangerous areas and necessary instruction confirmations. The first threshold Cth1 and the second threshold Cth2 are collectively referred to as cognitive load thresholds, which satisfy 0<Cth1<Cth2<1. Since the cognitive load C(t) is output via the Sigmoid function in formula (4), its value range is [0,1]. As an embodiment of the present invention, Cth1 is 0.6, and Cth2 is 0.8. The method for determining the cognitive load thresholds is as follows: in the offline calibration stage, the operator's cognitive load C(t) and operation performance indicators are collected synchronously, wherein the operation performance indicators include target interpretation accuracy, instruction response time and misoperation rate; the C(t) value corresponding to the inflection point where the operation performance indicators begin to decrease significantly as C(t) increases is taken as the first threshold, and the C(t) value corresponding to the significant increase of the misoperation rate is taken as the second threshold; in the system operation stage, the above thresholds are individually calibrated according to the operator's historical interaction data, for example, the 70th percentile and the 90th percentile of the operator's historical cognitive load distribution are respectively taken as the individual first threshold and second threshold. To avoid frequent switching of display modes caused by fluctuations of cognitive load near the thresholds, a hysteresis band Δ is set for both the first threshold and the second threshold, and the value range of Δ is 0.03 to 0.10, that is, the corresponding display mode is switched to when the cognitive load rises above the threshold, and the display mode is exited only when it drops below the threshold minus Δ.

[0040] Step S420, smooth switching and safety confirmation. To avoid frequent interface jumps, the policy engine sets a change penalty term for the display control actions at two consecutive moments, and executes the layout switching only when the benefit exceeds the threshold. For key instructions involving target locking, alarm elimination, task mode switching, etc., when the intention confidence R(t) is lower than the set threshold or the fatigue state F(t) rises, the system automatically enters a secondary confirmation mode to reduce the risk of misoperation.

[0041] Step S430, cognitive feedback closed loop. After the display interface executes the control action, the operator's new gaze point, operation result and command response time are fed back to step S300, forming a closed loop of "display adjustment - state change - strategy re-adjustment". Please refer to Figure 2 , Figure 2 which is a schematic diagram of the system structure of the cognitive closed-loop sonar high-dimensional display.

[0042] Step S500, executing the display control action to update the sonar high-dimensional display interface, and performing fatigue warning.

[0043] Step S510: Interactive command execution. Specifically, based on the interactive intent U(t) output in step S300 and the command confirmation mode determined in step S400, operations such as target selection, viewpoint switching, section plane adjustment, timeline dragging, target marking, alarm confirmation, and history playback are executed. When both voice and gesture point to the same target, the system increases the confidence level of the voice command; when conflicts occur between different modalities, the system pauses execution and requests confirmation from the operator using a candidate menu.

[0044] Step S520, fatigue warning, specifically: determine the fatigue level based on fatigue state, continuous working time, blinking frequency, gaze dispersion and number of misoperations; when the fatigue level reaches the warning threshold, reduce interface brightness stimulation, reduce low-priority dynamic effects, issue a rest reminder or synchronize key alarms to the collaborating operator.

[0045] As a specific embodiment of the present invention, the fatigue level G(t) is determined by the comprehensive fatigue score S(t): S(t) = κ1·F(t) + κ2·ê1(t) + κ3·ê2(t) + κ4·ê3(t) + κ5·ê4(t).

[0046] Wherein, ê1(t) is the normalized continuous working time, which is linearly normalized to the full scale with the rated continuous duty time (e.g., 4 h); ê2(t) is the normalized blink frequency deviation, which is the individual baseline obtained by counting the blink frequency in the first 10 minutes after the operator starts the operation, and is 1 when the blink frequency increases by 50% or more relative to the baseline; ê3(t) is the normalized fixation dispersion, which is linearly normalized to the ratio of the standard deviation of the spatial distribution of fixation points in the current time window to the individual baseline, and is 1 when the ratio reaches 1.5 or more; ê4(t) is the normalized number of misoperations, which is 1 when it occurs 3 or more times every 10 minutes; κ1~κ5 are the weight coefficients of each component, and the value range is [0,1] and satisfies κ1+κ2+κ3+κ4+κ5=1. As an example, κ=[0.40, 0.20, 0.15, 0.10, 0.15]. When S(t) falls within [0,0.4), [0.4,0.6), [0.6,0.8), and [0.8,1] respectively, the fatigue level is determined to be from level one to level four.

[0047] The classification range and weighting coefficients for each level can be individually calibrated based on the operator's historical interaction data. In scenarios where sensors are limited and gaze dispersion or blink frequency cannot be obtained, the weights of the corresponding components can be reset to zero and the remaining weights can be renormalized before continuing the calculation.

[0048] As a specific embodiment of the present invention, the value of the warning threshold can be individually calibrated based on the operator's historical interaction data. For example, the 70th percentile of the operator's fatigue level can be used as the individualized warning threshold. Preferably, to avoid frequent switching of display modes due to fluctuations in fatigue level around the threshold, a hysteresis band Δ is set for the warning threshold. The value of Δ ranges from 0.03 to 0.10, meaning that the corresponding mode is switched when the warning threshold rises above the threshold and exits the corresponding mode only when it falls below the threshold minus Δ.

[0049] Step S530, historical playback and review, specifically: the display control actions, operator state estimation results, actual instructions and task results are associated and stored for post-event review, model calibration and individualized threshold update.

[0050] In step S600, the cognitive state estimation model and the display strategy model are lightweighted at the edge, and the individualized parameters are updated based on the operator's historical interaction data.

[0051] Step S610 involves lightweighting the cognitive state estimation model and display strategy model at the edge. Specifically, this involves pruning, quantizing, or distilling the cognitive state estimation model and display strategy model to enable them to run with low latency on shipborne terminals, unmanned underwater vehicle control units, or portable shore-based devices. For tasks requiring high real-time performance, the strategy engine can pre-generate a task-load-display action lookup table and quickly retrieve and execute it at the edge. That is, at the edge, the corresponding display action is directly retrieved and executed based on the current task context and cognitive load.

[0052] Step S620 involves updating individualized parameters based on the operator's historical interaction data. Specifically, this includes: locally analyzing different operators' gaze habits, command speed, fatigue threshold, and error types; and individually calibrating the weight parameters w1 to w6, cognitive load threshold, and display load upper limit Bmax in the acoustic saliency calculation. Calibration data is preferentially stored locally, with only anonymized statistical parameters or model increments uploaded to avoid leakage of original operational data.

[0053] As another embodiment of the present invention, the individualized parameters can be updated using local incremental learning, periodic offline calibration, or manual parameter configuration. In application scenarios with low privacy requirements, the anonymized logs can also be uploaded to the central terminal for unified model optimization before being distributed.

[0054] Step S630, adaptive resource allocation: Specifically, the edge dynamically allocates computing resources among high-precision rendering, cognitive state estimation, historical playback, and alarm inference based on remaining computing power, current task stage, and cognitive load level. When a task is urgent or the cognitive load increases, priority is given to ensuring the frame rate and response time of the target annotation, alarm display, and command confirmation links.

[0055] (9) Where k is the category number of the edge computing task, k=1,2,…,K, and K is the total number of task categories; as an embodiment of the present invention, the computing tasks include four categories: high-precision rendering, cognitive state estimation, historical playback, and alarm reasoning, in which case K=4; rk is the edge computing resource allocated to the k-th type of task; r=[r1,r2,…,rK] T Let r* be the resource allocation vector, representing the optimal resource allocation scheme that minimizes the objective function in formula (9); Dk is the processing latency of the task, and Ek is the energy consumption or computing power overhead; Dk(rk) is the processing latency of the k-th type of task under resource rk, and Ek(rk) is the energy consumption or computing power overhead of the k-th type of task under resource rk; η is the weight coefficient of the energy consumption or computing power overhead term, η>0, used to balance processing latency and energy consumption. As an example, the value of η ranges from 0.1 to 1.0; Redge is the total amount of computing resources available to the edge node, Lkey is the latency of the critical display and control link, and Lmax is the maximum allowed latency of the critical link. This constraint ensures that safety-critical tasks such as alarms, target labeling, and command confirmation are prioritized when resources are limited.

[0056] Please see Figure 3 , Figure 3 This is another flowchart of a high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation.

Claims

1. A high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state, characterized in that, The method includes the following steps: Acquire high-dimensional display elements output by the sonar front-end processing module, and abstract these high-dimensional display elements into a unified display object; The task context is constructed by combining the current task stage and the operator's operation goal, and the acoustic saliency and display priority of each displayed object are calculated; Collect multimodal data of the operator, and estimate the operator's cognitive load, fatigue state, interaction intention and intention confidence in real time based on the multimodal data; The display priority, task context, cognitive load, fatigue state, interaction intent, and intent confidence are input into the closed-loop display strategy engine to generate display control actions; The display control action is executed to update the sonar high-dimensional display interface and provide fatigue warning; The cognitive state estimation model and the display strategy model are lightweighted at the edge, and the individualized parameters are updated based on the operator's historical interaction data.

2. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 1, characterized in that, The calculation of the acoustic saliency and display priority of each display object includes: Obtain the echo intensity, spectral concentration, trajectory anomaly, confidence level, task relevance, and occlusion penalty of the displayed object; The formula for calculating the acoustic saliency of the displayed object is: Ai=w1×qi+w2×ri+w3×di+w4×ui+w5×hi-w6×bi (1); The formula for calculating the display priority of a display object is: (2); Where Ai represents the acoustic saliency of the i-th display object; represents the normalized display priority of the i-th display object; qi represents the echo intensity of the i-th display object; ri represents the spectral concentration of the i-th display object; di represents the trajectory anomaly of the i-th display object; ui represents the confidence level of the i-th display object; hi represents the task relevance of the i-th display object; bi represents the occlusion penalty of the i-th display object; τ is the temperature coefficient; m is the total number of display objects in the current time window; Aj represents the acoustic saliency of the j-th display object; w1, w2, w3, w4, w5, and w6 are weight parameters; each weight parameter is dynamically adjusted according to the task context.

3. The high-dimensional adaptive display and control method for sonar images based on cognitive state closed-loop regulation according to claim 1, characterized in that, The real-time estimation of the operator's cognitive load, fatigue state, interaction intention, and intention confidence based on the multimodal data includes: The multimodal data includes voice commands, gestures, gaze points, pupil changes, blink frequency, EEG rhythm characteristics, mouse / knob / touch operations, and command response time; Using sonar display frames as a reference, multimodal data are mapped into a unified time window to extract fixation time, scan rate, pupil diameter change rate, blink interval, voice command confidence, gesture completion time, number of misoperations and EEG frequency band energy to obtain the operator's state feature vector; The state feature vector, the task context, and the most recent multiple interaction actions are input into a lightweight temporal attention model or a gated loop model, and the operator's cognitive load, fatigue state, interaction intention, and intention confidence are output.

4. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 3, characterized in that, The step of inputting the state feature vector, the task context, and the most recent multiple interaction actions into a lightweight temporal attention model or a gated recurrent model, and outputting the operator's cognitive load, fatigue state, interaction intent, and intent confidence, includes: [C(t), F(t), U(t), R(t)] = fθ(H(t), M(t), A(t)) (3) C ( t ) = σ ( αc T H(t)+βc T M(t)+γc T A(t)+bc) (4) F(t)=ρF(t-1)+(1-ρ)σ(αf T H(t)+βf T E(t)+bf) (5) Where C(t) represents cognitive load, F(t) represents fatigue state, U(t) represents interaction intent, R(t) represents intent confidence, Ā(t) is the saliency statistic of the currently displayed object, E(t) is the fatigue feature vector composed of continuous working time, blink frequency, and number of misoperations, σ is the Sigmoid function, ρ is the time memory coefficient of fatigue state, M(t) is the task context, A(t) is the acoustic saliency set of the currently displayed object, and fθ is the cognitive state estimation model deployed at the edge; the superscript ᵀ indicates vector transpose; αc, βc, and γc are the weight vectors corresponding to the state feature vector H(t), task context vector M(t), and saliency statistic Ā(t) in cognitive load estimation, respectively; bc is the bias term of cognitive load estimation, which is a scalar; αf and βf are the weight vectors corresponding to the state feature vector H(t) and fatigue feature vector E(t) in fatigue state estimation, respectively; bf is the bias term of fatigue state estimation.

5. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 1, characterized in that, The step involves inputting the display priority, task context, cognitive load, fatigue state, interaction intent, and intent confidence into the closed-loop display strategy engine to generate display control actions, including: The display control actions include: layer on / off, object aggregation granularity, volume data sampling density, time window length, target annotation intensity, alarm sound level, viewing focus, color scheme, and command confirmation mode; at = π(pi, M(t), C(t), F(t), U(t), R(t)) (6) (7) (8) Where, at is the display control action output at time t, π represents the display strategy function implemented by the closed-loop display strategy engine, i is the index of the display object, i=1,2,…,m, m is the total number of display objects in the current time window, Ω is the set of display control actions that the display control terminal can execute at the current time, a is any feasible display control action, π* represents the optimal display strategy that maximizes the objective function in formula (7); pi is the normalized display priority of the i-th display object; Vi(a) represents the control action a for the i-th display object. The visibility benefit is defined as follows: Ld(a) is the jump penalty between two adjacent interface actions, Lc(a) is the cognitive load penalty, and Le(a) is the edge computing overhead penalty; Tkey(a) represents the critical instruction link latency, Tmax is the maximum allowed response time, C(t) represents the cognitive load, R(t) is the intent confidence, Bmax is the display load limit, bi(a) represents the information load of control action a on the i-th display object, and λ1, λ2, and λ3 are the weight coefficients of the jump penalty, cognitive load penalty, and edge computing overhead penalty, respectively. Set a display load limit constraint. When the cognitive load exceeds the first threshold, reduce the display of low-priority noise layers and expand the labeling of high-priority targets. When the cognitive load exceeds the second threshold, switch to simplified display and control mode, retaining only targets, tracks, dangerous areas, and necessary command confirmations.

6. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 1, characterized in that, The fatigue warning system includes: The fatigue level is determined based on fatigue status, continuous working duration, blinking frequency, gaze dispersion, and number of misoperations. When the fatigue level reaches the warning threshold, reduce the brightness of the interface, decrease low-priority dynamic effects, issue a rest reminder, or synchronize key alarms to the collaborating operators.

7. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 1, characterized in that, The lightweighting of the cognitive state estimation model and the display strategy model at the edge includes: Pruning, quantization, or distillation are performed on the cognitive state estimation model and the explicit strategy model; For tasks requiring real-time performance, a task-load-display action lookup table is pre-generated, and the corresponding display action is directly retrieved and executed at the edge based on the current task context and cognitive load.

8. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 1, characterized in that, The step of updating individualized parameters based on the operator's historical interaction data includes: Locally analyze different operators' gaze habits, command speed, fatigue threshold, and types of misoperations; Individualized calibration of weighting parameters, cognitive load threshold, and display load upper limit in acoustic saliency calculation; Calibration data is stored locally first, and only anonymized statistical parameters or model increments are uploaded to the central terminal.

9. The high-dimensional adaptive display and control method for sonar images based on closed-loop control of cognitive state as described in claim 1, characterized in that, After updating the individualized parameters based on the operator's historical interaction data, the method further includes: At the edge, computing resources are dynamically allocated among high-precision rendering, cognitive state estimation, historical playback, and alarm inference based on remaining computing power, current task stage, and cognitive load level. When tasks are urgent or cognitive load increases, priority should be given to ensuring the frame rate and response time of the target labeling, alarm display and command confirmation links, while limiting the resource consumption of non-critical tasks.