Modular multi-modal situation awareness ballistic smart helmet system

Abstract

The application discloses a modular multi-modal situation awareness bulletproof intelligent helmet system, comprising a multi-source heterogeneous positioning and networking module, an eye movement tracking module, a nine-axis inertial unit, a transparent mask and a processor on the helmet. The processor is configured to: under the condition of no reference point, a global coordinate system is constructed by graph optimization through inter-node relative ranging; when UWB is non-line-of-sight, the Kalman innovation covariance is adaptively increased according to signal strength, pulse width and ranging residual error to reduce the ranging fusion weight; when the network is congested, the transmission power is reduced according to the node depth classification, and the nodes with small depth keep high power to maintain the backbone connectivity; the corresponding information and the sound in the real-time gaze direction are enhanced; the eye ball coordinates and the helmet attitude angle are fused to compensate for the gaze point prediction error caused by the rapid rotation of the head. The multi-node cooperative positioning accuracy, the self-organizing network energy efficiency and the backbone connectivity are improved, and the problems of insufficient situation awareness under the conditions of no infrastructure, signal shielding, topology change and rapid head rotation are solved.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a modular multimodal situational awareness bulletproof smart helmet system. Background Technology

[0002] As a critical wearable device in complex environments such as individual combat, fire rescue, and field exploration, the situational awareness capability of smart helmets directly impacts mission efficiency and personnel safety. In existing technologies, bulletproof helmets primarily focus on physical protection, but in recent years they have gradually evolved towards intelligent design, integrating positioning, communication, and display modules. However, in scenarios without GPS signals or base station coverage, such as indoors, tunnels, and forests, traditional positioning methods relying on absolute reference points become completely ineffective. To address this issue, some existing solutions employ ultra-wideband (UWB) technology to achieve relative ranging between nodes and share data through ad-hoc wireless networks. For example, existing research has disclosed a multi-robot relative positioning method based on UWB ranging, as well as a voice communication system based on mobile ad-hoc networks. However, the above-mentioned solutions have significant shortcomings in practical applications: First, the ranging error of UWB signals increases sharply in non-line-of-sight (NLOS) environments such as human body obstruction and wall blockage. Existing methods mostly use simple line-of-sight / non-line-of-sight binary judgment, lacking a refined measurement of the degree of non-line-of-sight and its adaptive weight adjustment in fusion positioning, resulting in a serious decrease in positioning accuracy. Second, the power control of ad hoc network communication is usually based on a fixed distance threshold or received signal strength, without considering the impact of node depth (hop count) and congestion status on overall connectivity, which can easily lead to backbone link interruption or energy waste at edge nodes. Third, after a multi-node subnet splits due to communication interruption, there is a lack of a reliable coordinate system merging mechanism when they meet again. Existing technologies mostly rely on manual calibration or the assumption of a common reference point, making it difficult to achieve automatic, low-error coordinate system unification.

[0003] On the other hand, to reduce the cognitive load on wearers, some smart helmets have attempted to introduce augmented reality display technology, overlaying navigation information or teammate locations onto the face mask. In existing solutions, the displayed content is mostly a fixed layout or highlighted cues based on simple rules, failing to dynamically couple with the wearer's visual attention. For example, some literature discloses methods for switching displayed content using eye tracking, but these typically only control the on / off state of a single information source, failing to achieve hierarchical enhancement of multi-target information and synchronous gain of ambient sound. Furthermore, during rapid head rotation, the image processing delay of the eye-tracking camera leads to lag in gaze prediction. Existing compensation methods mostly employ Kalman filtering based on pure eye movement models, ignoring the coupling effect of helmet posture changes on gaze, thus limiting the compensation effect. In summary, existing technologies still have gaps in areas such as dynamic collaborative positioning without infrastructure support, adaptive fusion against non-line-of-sight interference, network power management under dynamic topology, and attention-oriented multimodal information enhancement. A modular smart helmet system capable of comprehensively addressing these issues is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide a modular, multimodal situational awareness bulletproof smart helmet system. By optimizing base station-free coordinate construction through fusion graph optimization, adjusting covariance of multi-parameter adaptive Kalman filtering in non-line-of-sight states, hierarchical power control based on network depth, and collaborative compensation of eye movement and helmet posture, it improves the accuracy and robustness of multi-node collaborative positioning in complex environments, the energy efficiency and backbone connectivity of self-organizing network communication, and the real-time performance of attention-oriented information enhancement. It solves the problem of insufficient situational awareness capabilities in existing technologies when there is no infrastructure support, signal obstruction, dynamic changes in network topology, and rapid head rotation.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a modular multimodal situational awareness bulletproof smart helmet system, comprising: The helmet body includes a multi-source heterogeneous positioning and networking module, an eye-tracking module, a nine-axis inertial measurement unit, and a transparent face mask display; and a processor configured to perform the following operations: Multi-node cooperative positioning and ad hoc network communication include: constructing a global coordinate system using graph optimization algorithms based on the relative ranging values ​​between nodes when there is no absolute reference point; when a UWB signal is received and it is determined to be in a non-line-of-sight state, the non-line-of-sight state is comprehensively determined based on the received signal strength, impulse response width, and ranging residual, and the Kalman filter innovation covariance is adaptively increased to reduce the weight of the ranging value in the fusion; when the network is congested, the transmission power is reduced in stages according to the depth of each node in the network, with nodes at lower depths maintaining higher transmission power to maintain the connectivity of the backbone network; Based on the wearer's real-time gaze direction detected by the eye-tracking module, the information display in the corresponding direction is enhanced on the transparent mask display, and the ambient sound in that direction is simultaneously enhanced; When enhancing the ambient sound, the relative eye coordinates obtained by eye tracking are fused with the helmet attitude angle obtained by the nine-axis inertial measurement unit to compensate for the gaze point prediction error caused by rapid head rotation.

[0006] Furthermore, in multi-node cooperative localization, the processor is also configured to: when two or more subnets meet again after splitting due to communication interruption, perform coordinate system merging based on graph matching, specifically including: detecting overlapping nodes; constructing a least-squares similarity transformation between the coordinate systems of the two subnets using the relative positional relationship of the overlapping nodes; eliminating mismatched node pairs through a random sampling consensus algorithm; calculating the optimal rotation and translation matrix using the four-point method; and merging is performed only when the median of the interior point residuals is less than a preset threshold, otherwise triggering the re-initialization of global graph optimization.

[0007] Furthermore, the coordinate system merging based on graph matching also includes: initial alignment based on graph Laplacian eigenvectors, that is, before using the random sampling consensus algorithm, extracting the adjacency matrices of overlapping nodes in the two subnets respectively, calculating the second small eigenvector of the graph Laplacian matrix of each adjacency matrix, and determining the preliminary correspondence of overlapping nodes by matching the second small eigenvectors.

[0008] Furthermore, the processor is also configured to: perform global consistency optimization after the coordinate system is merged, that is, the merged global coordinate system is refined through pose graph optimization, wherein the pose node of each node is connected to the ranging edge of its neighboring nodes; add the transformation matrix estimated by the random sampling consensus algorithm during the merging process as a strong prior factor to the pose graph, and perform real-time optimization using a gradient descent and Gauss-Newton hybrid solver.

[0009] Furthermore, the processor is configured in congestion management of ad hoc network communication to perform hybrid hierarchical power control based on channel occupancy and node mobility, including: Each node periodically measures the channel busy-idle ratio and calculates its own instantaneous velocity using a nine-axis inertial measurement unit. When the channel occupancy rate exceeds the first threshold, the node network depth is used as the classification criterion. When the channel occupancy rate is lower than the second threshold and the node's instantaneous velocity exceeds the velocity threshold, the transmit power is automatically increased. The depth classification uses a nonlinear attenuation function, so that the transmit power attenuation rate of nodes with smaller depths is lower than that of nodes with larger depths.

[0010] Furthermore, the nonlinear decay function is expressed as: ; in, The network depth of the node. At maximum transmission power, This is an attenuation coefficient that is dynamically adjusted based on channel occupancy; the higher the channel occupancy, the more attenuation occurs. The larger the value; When the instantaneous velocity of a node exceeds a velocity threshold, Multiply by a factor less than 1.

[0011] Furthermore, the processor is also configured to: adaptively adjust the lower limit of the transmit power based on depth and speed, that is, the minimum transmit power of each node is dynamically calculated according to its current depth and instantaneous speed; nodes with greater depth and slower speed are allowed to be reduced to a smaller power value; nodes with smaller depth or faster speed have their minimum power clamped at a higher level.

[0012] Furthermore, the processor, when performing eye-tracking selective enhancement, is also configured to: construct a gaze-driven dynamic information priority queue, including: The position information of all teammates within the field of view of the transparent mask display is sorted according to the angle difference with the gaze direction; only teammates whose angle difference is less than the dynamic threshold are highlighted; the dynamic threshold is adaptively adjusted according to the dwell time of the current gaze point and the movement speed of the gaze point, the longer the dwell time or the slower the movement speed, the smaller the dynamic threshold; the information in the non-gaze area is rendered with gradient transparency, and the transparency is proportional to the angle difference.

[0013] Furthermore, the adaptive adjustment rule for the dynamic threshold is as follows: ; in, For the maximum angular difference, The duration of continuous fixation at the current point of gaze. The velocity of the gaze point movement. The preset sensitivity coefficient; when When it increases Approaching ,when When θ increases th Decrease.

[0014] Furthermore, in the gradient transparency rendering, the transparency α is related to the angle difference. The relationship uses a sigmoid function: ; in, The semi-transparency threshold angle, This is the transition steepness coefficient; at the fan-shaped boundary near the gaze direction, the transparency gradient is gentle, while in the region far from the gaze direction, the transparency rapidly approaches its maximum value.

[0015] The above-described technical solutions of the embodiments of the present invention have the following beneficial technical effects: 1. By automatically constructing a global coordinate system based on graph optimization under conditions without absolute reference points, adaptively increasing the Kalman filter innovation covariance based on received signal strength, impulse response width, and ranging residual in non-line-of-sight conditions to reduce the fusion weight of unreliable ranging values, and reducing the transmission power according to the node network depth and using random sampling consistency to eliminate mismatched node pairs when they meet again after subnet splitting, this application significantly improves the initial convergence speed and continuous tracking accuracy of multi-node cooperative positioning in complex environments such as indoors, tunnels, and forests without GPS or base station coverage. It effectively suppresses positioning drift caused by non-line-of-sight interference and ensures the connectivity of the backbone link when the network is congested. This solves the core contradiction between positioning robustness and self-organizing network energy efficiency that is difficult to balance in existing technologies. 2. By constructing a gaze-driven dynamic information priority queue, the position information of teammates within the field of view of the transparent mask display is sorted according to the angle difference with the gaze direction, and only teammates whose angle difference is less than the dynamic threshold are highlighted and magnified. At the same time, the information in the non-gaze area is rendered with gradual transparency, and the ambient sound in the gaze direction is simultaneously enhanced through bone conduction headphones. This application realizes the dynamic tracking of the wearer's visual attention by information display and auditory enhancement, avoids the distraction problem caused by information overload or fixed display in traditional helmets, significantly reduces the cognitive load of situational awareness, and solves the technical problem of difficulty in quickly and effectively acquiring multi-source information in high-pressure environments such as battlefields or rescue. 3. By fusing the relative eye coordinates obtained from eye tracking with the helmet attitude angles obtained from a nine-axis inertial measurement unit, the state variables of the Kalman filter simultaneously include both eye angular velocity and helmet angular velocity. Predictive compensation is used to make the highlighted display area lead the actual gaze point. This application eliminates the gaze point prediction lag error caused by rapid head rotation, improving the real-time performance and accuracy of eye-tracking interaction during vigorous movement or rapid head turns. This ensures the synchronization accuracy of attention enhancement display and sound gain, solving the display jitter or response delay problem caused by neglecting helmet attitude coupling in existing eye tracking systems in dynamic helmet scenarios. This further enhances the user experience and operational reliability of the overall situational awareness system. Attached Figure Description

[0016] Figure 1 This is a block diagram of the system modules provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the core functional flow of the processor provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0018] The modular multimodal situational awareness bulletproof smart helmet system provided in this invention is mainly applied in complex environments such as individual combat, fire rescue, field exploration, and urban counter-terrorism, where there is no fixed infrastructure support or GPS signal rejection. In these scenarios, wearers often need to coordinate with teammates under highly dynamic, highly obscured, and highly interfered conditions, maintaining physical protection while simultaneously acquiring real-time information on their own and their teammates' positions and environmental threats, and rapidly processing multi-source data to avoid cognitive overload. Traditional bulletproof helmets only provide passive protection, while existing helmet solutions integrating communication or positioning modules suffer from numerous drawbacks, such as positioning drift in non-line-of-sight environments, the contradiction between self-organizing network power consumption and connectivity, and delays in eye-tracking interaction. To address this, this invention proposes a modular multimodal situational awareness bulletproof smart helmet system that organically integrates three major functions: multi-node collaborative positioning and self-organizing network communication, eye-tracking-driven attention-selective enhanced display, and eye-helmet posture fusion compensation. Through a processor, it achieves graph-optimized base station-free coordinate construction, multi-parameter adaptive Kalman filter covariance adjustment in non-line-of-sight states, hierarchical power control based on network depth, and gaze-guided information and sound synchronization enhancement. Thus, under harsh conditions such as lack of infrastructure support, signal obstruction, dynamic changes in network topology, and rapid head rotation, it significantly improves the accuracy of multi-node relative positioning, the robustness of communication links, and the real-time performance and low cognitive load of human-computer interaction.

[0019] Please refer to Figure 1 and Figure 2 The modular multimodal situational awareness bulletproof smart helmet system of this invention specifically includes: a multi-source heterogeneous positioning and networking module, an eye-tracking module, a nine-axis inertial measurement unit, a transparent face mask display, and a processor, all installed on the helmet body. The multi-source heterogeneous positioning and networking module includes a UWB ranging submodule, a self-organizing network communication submodule, and optional other positioning units, used for relative distance measurement between nodes, multi-hop data relay, and dynamic power adjustment. The eye-tracking module typically consists of a miniature infrared camera and a light source, capturing the wearer's pupil position in real time to calculate the gaze direction. The nine-axis inertial measurement unit integrates a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, used to measure the helmet's instantaneous motion state and attitude angle. The transparent face mask display uses waveguide or freeform surface optics technology to overlay and display teammate location heatmaps, navigation indicators, and alarm information within the wearer's field of vision.

[0020] The graphical interface presented by the face mask display supports customizable configuration. Users can set the type, transparency, and layout priority of the displayed layers via host computer software (such as a tablet or command terminal). By default, the system displays three layers: the bottom layer is a semi-transparent electronic compass and speedometer; the middle layer is a heat map of teammates' positions (color depth indicates distance); and the top layer is a threat alarm icon (such as a flashing red border when a rapidly approaching object is detected). In addition, the display supports a night vision mode—when the ambient light sensor detects illuminance below 1 lux, it automatically switches to a green or red monochrome display to avoid impairing the wearer's dark adaptation ability. Rendering of all layers is handled by the processor's graphics processing unit (GPU, such as an ARM Mali-G78), with a frame rate of at least 30 frames per second, ensuring smooth dynamic updates without ghosting.

[0021] As an optional implementation, the UWB ranging submodule can use Decawave's DW3000 series chipset, operating in the 6.5 GHz to 8.0 GHz frequency band, supporting both dual-path ranging (TWR) and phase difference of arrival (PDOA) ranging modes. In TWR mode, the ranging frequency can reach up to 100 Hz, with a static accuracy better than 10 cm; in PDOA mode, it can simultaneously output the angle of arrival (AOA), with an angular resolution of approximately 3 degrees. The self-organizing network communication module can use a Mesh Wi-Fi chip based on the IEEE 802.11s standard, such as the Qualcomm QCA4020, or a LoRa spread spectrum communication chip (such as the SX1262) to reduce power consumption, allowing for flexible switching according to task requirements. The nine-axis inertial measurement unit can be either the BMI160 or ICM-20948, which integrates a digital motion processor (DMP) and can output calibrated quaternions and Euler angles, with a sampling rate up to 400 Hz. The eye-tracking module can utilize Pupil Labs' core components based on the pupil-corneal reflection method, consisting of two 80 Hz near-infrared cameras and four LED light sources, connected to the processor via a USB or MIPI interface. The transparent mask display can employ a waveguide solution (such as Lumus or DigiLens products) or a holographic thin-film solution, with a field of view of no less than 30 degrees and an adjustable brightness range of 100 to 3000 nits to adapt to different ambient lighting conditions.

[0022] To enhance the system's battlefield adaptability and ease of maintenance, the helmet features a standardized modular interface. The UWB ranging module, self-organizing network communication module, and nine-axis inertial measurement unit are all installed as independent, pluggable modules within this interface, supporting hot-swappable replacement. When a module fails or requires an upgrade, the wearer can quickly replace it without tools, ensuring mission continuity. Furthermore, the standardized interface allows for the selection of positioning or communication modules with different performance levels according to mission requirements. For example, a high-power UWB module can be replaced in heavily obscured environments, or a narrowband communication module can be used for low-power applications, achieving a high degree of system configurability.

[0023] As an alternative implementation, the processor employs a heterogeneous multi-core architecture, including a high-performance ARM Cortex-A series core (for complex operations such as graph optimization and Kalman filtering) and a low-power Cortex-M series core (for UWB ranging, IMU data acquisition, and real-time response of the ad hoc network protocol stack). The two cores communicate via shared memory and support dynamic frequency adjustment. When the system is in standby or stationary mode, the high-performance core enters deep sleep mode, while only the low-power core maintains the minimum network heartbeat and inertial sensor data cache, thereby reducing the overall power consumption to below 200 milliwatts. When node movement or network data mutation is detected, the high-performance core quickly wakes up (wake-up time less than 1 millisecond) to complete positioning calculation and display refresh. Furthermore, the helmet body has a built-in replaceable lithium polymer battery pack (5000 mAh capacity, 7.4 V rated voltage) that supports contact charging and wireless charging (Qi standard). A battery management chip (such as BQ25601) monitors the remaining battery level in real time and provides a low-battery warning via a transparent face mask display.

[0024] The processor is configured to perform the following operations: multi-node collaborative localization and ad hoc network communication, including: constructing a global coordinate system using graph optimization algorithms based on the relative ranging values ​​between nodes when no absolute reference point is available; when a UWB signal is received and it is determined to be in a non-line-of-sight state, comprehensively determining the non-line-of-sight state based on the received signal strength, impulse response width, and ranging residual, and adaptively increasing the Kalman filter innovation covariance to reduce the weight of the ranging value in the fusion; when network congestion occurs, reducing the transmission power according to the depth of each node in the network, with nodes at lower depths maintaining higher transmission power to maintain the connectivity of the backbone network. The processor is also configured to perform the following operations: enhance the display of information in the corresponding direction on the transparent mask display based on the wearer's real-time gaze direction detected by the eye-tracking module, and simultaneously enhance the ambient sound in that direction. The processor is also configured to perform the following operations: when enhancing ambient sound, fusing the relative eye coordinates obtained from eye-tracking with the helmet attitude angle obtained from the nine-axis inertial measurement unit to compensate for gaze prediction errors caused by rapid head rotation.

[0025] The processor is the core of the entire system's control and computation, configured to perform three main categories of key operations. The first category is multi-node collaborative positioning and ad hoc network communication: Under initial conditions without any absolute reference points (such as GPS satellites or ground base stations), the processor automatically constructs a global relative coordinate system using graph optimization algorithms based solely on the relative ranging values ​​between nodes, and continuously converges the coordinates. When a UWB signal is received and it is determined to be in a non-line-of-sight (NLS) state, the processor comprehensively determines the degree of NLS based on the received signal strength, impulse response width, and ranging residual, and adaptively increases the Kalman filter innovation covariance, thereby reducing the weight of the unreliable ranging value in the fusion positioning and suppressing positioning drift. When congestion occurs in the ad hoc network communication, the processor performs graded transmission power control according to the depth (hop count) of each node in the network. Nodes with lower depths (closer to the network's logical center) maintain higher transmission power to maintain backbone connectivity, while nodes with higher depths appropriately reduce power to minimize collisions and energy consumption. The second type is selective enhancement operation driven by eye tracking: Based on the wearer's gaze direction detected in real time by the eye tracking module, the processor dynamically identifies the corresponding fan-shaped area on the transparent mask display and only highlights and magnifies the teammate's position information within that area. Simultaneously, ambient sound from that direction is amplified through bone conduction headphones, while the brightness of information in non-gaze areas is reduced and the sound volume remains the original, thus achieving an attention-oriented "narrowed focus, wide sweeping" effect. The third type is eye-helmet posture fusion compensation operation: During the enhancement of ambient sound and display information, the processor fuses the relative eye coordinates obtained from eye tracking with the helmet posture angle obtained from the nine-axis inertial measurement unit. This ensures that the Kalman filter state quantity predicted for the gaze point includes both eye angular velocity and helmet angular velocity, effectively compensating for the prediction delay caused by rapid head rotation. This ensures that the highlighted display area is always ahead of the actual gaze point by approximately tens of milliseconds, eliminating display lag. Through the multimodal collaborative computing of the processor described above, this system can provide wearers with high-precision, low-latency, and low-cognitive-load situational awareness capabilities in dynamic, occluded, and infrastructure-free extreme environments.

[0026] Furthermore, in multi-node cooperative localization, the processor is also configured to: when two or more subnets meet again after splitting due to communication interruption, perform coordinate system merging based on graph matching, specifically including: detecting overlapping nodes; constructing a least-squares similarity transformation between the coordinate systems of the two subnets using the relative positional relationship of the overlapping nodes; eliminating mismatched node pairs through a random sampling consensus algorithm; calculating the optimal rotation and translation matrix using the four-point method; and merging is performed only when the median of the interior point residuals is less than a preset threshold, otherwise triggering the re-initialization of global graph optimization.

[0027] After multiple helmet nodes split into two or more subnets due to communication interruption, the processor automatically triggers a graph-matching-based coordinate system merging process when these subnets re-enter each other's communication range. This process first detects overlapping nodes (i.e., the same physical node simultaneously identified by two subnets) that exist between the subnets. Then, using the relative positions of these overlapping nodes in their respective subnet coordinate systems, a least-squares similarity transformation is constructed between the two subnet coordinate systems to obtain a preliminary coordinate correspondence. To eliminate erroneous matching pairs caused by ranging errors or environmental interference, the processor further employs the Random Sample Consensus (RANSAC) algorithm to iteratively filter overlapping node pairs, removing erroneous matching points that deviate from the mainstream transformation. Based on the filtered reliable interior points, the optimal rotation and translation matrix between the two coordinate systems is precisely solved using the Four-Point Congruent Set method.

[0028] After calculating the transformation matrix, the processor checks if the median of the interior point residuals is below a preset tolerance threshold. If it is below the threshold, the merging quality is considered satisfactory, and coordinate system merging is formally executed, unifying the two subnets into the same global reference frame. If the median residual exceeds the threshold, it indicates that the current matching result is unreliable. To avoid positioning jumps caused by erroneous merging, the processor triggers a re-initialization of global graph optimization, i.e., abandoning the current merging result and reconstructing a unified coordinate system from the original ranging data. Through this mechanism, the system can automatically and robustly complete subnet coordinate fusion under dynamic topology changes without manual intervention or reliance on external benchmarks.

[0029] Furthermore, the coordinate system merging based on graph matching also includes: initial alignment based on graph Laplacian eigenvectors, that is, before using the random sampling consensus algorithm, extracting the adjacency matrices of overlapping nodes in the two subnets respectively, calculating the second smallest eigenvector of the graph Laplacian matrix of each adjacency matrix, and determining the preliminary correspondence of overlapping nodes by matching the second smallest eigenvector.

[0030] Before merging coordinate systems based on Random Sample Consensus (RANSAC) and the four-point method, this system introduces an initial alignment method based on graph Laplacian eigenvectors to quickly determine the preliminary correspondence between overlapping nodes, thereby improving the efficiency and accuracy of subsequent matching. Specifically, for two subnets to be merged, the processor extracts the topological connectivity between overlapping nodes within each subnet, constructing an adjacency matrix for each subnet. This matrix reflects the relative ranging connectivity between overlapping nodes. Subsequently, the processor calculates the graph Laplacian matrix corresponding to each adjacency matrix and extracts the second smallest eigenvector (i.e., the Fiedler vector) of this Laplacian matrix. This vector maps nodes in the graph to a low-dimensional embedding space according to their topological positions, ensuring that nodes with similar structures have similar coordinate values ​​in the embedding space. By comparing the Fiedler vectors of the two subnets, the processor can quickly determine the preliminary correspondence between overlapping nodes, i.e., associate the same physical node in different subnets through the matching degree of vector values. This initial alignment step effectively reduces the number of iterations required for random sampling in the subsequent RANSAC algorithm and lowers the probability of incorrect matching caused by ranging noise or topological sparsity, providing a reliable initial assumption for calculating the optimal rotation and translation matrix using the four-point method.

[0031] Furthermore, the processor is also configured to: perform global consistency optimization after the coordinate system is merged, that is, the merged global coordinate system is refined through pose graph optimization, wherein the pose node of each node is connected to the ranging edge of its neighboring nodes; add the transformation matrix estimated by the random sampling consensus algorithm during the merging process as a strong prior factor to the pose graph, and perform real-time optimization using a gradient descent and Gauss-Newton hybrid solver.

[0032] After completing the coordinate system merging based on graph matching, the processor further performs global consistency optimization to eliminate residual ranging errors and local registration biases from the merging process. Specifically, the merged global coordinate system is modeled as a pose graph, where each node corresponds to the position and pose of a helmet, and edges between nodes connect adjacent nodes with direct ranging relationships. The constraint of the edges is the relative distance measured by UWB. Based on this, the processor adds the optimal rotation and translation matrix estimated by the Random Sample Consensus (RANSAC) algorithm during the coordinate merging stage as a strong prior factor to the pose graph. This prior factor constrains the relative pose relationship between the two subnets after merging, preventing drift during optimization. Subsequently, a gradient descent and Gauss-Newton hybrid solver is used to perform real-time iterative optimization of the pose graph. Gradient descent is used to quickly approach the optimal region, while Gauss-Newton is used for accurate convergence, thus obtaining globally consistent coordinate estimates while ensuring computational efficiency. Through the above global consistency optimization, the system can effectively suppress the accumulation of local ranging noise, ensuring that the positions of all nodes remain consistent throughout the entire network, further improving the accuracy and robustness of cooperative localization.

[0033] Furthermore, the processor is configured in congestion management for ad hoc network communication to perform hybrid hierarchical power control based on channel occupancy and node mobility, including: Each node periodically measures the channel busy-idle ratio and calculates its own instantaneous velocity using a nine-axis inertial measurement unit. When the channel occupancy rate exceeds the first threshold, the node network depth is used as the classification criterion. When the channel occupancy rate is lower than the second threshold and the node's instantaneous velocity exceeds the velocity threshold, the transmit power is automatically increased. The depth classification uses a nonlinear attenuation function, so that the transmit power attenuation rate of nodes with smaller depths is lower than that of nodes with larger depths.

[0034] In the congestion management process of ad hoc network communication, the processor executes a hybrid hierarchical power control strategy based on channel occupancy and node mobility. Specifically, each network node periodically measures the current busy-to-idle ratio of the channel (i.e., channel occupancy) and calculates its instantaneous velocity using a nine-axis inertial measurement unit. When the channel occupancy exceeds a preset first threshold, the network is determined to be in a congested state. At this time, the node's depth (hop count) in the network is used as the main basis for transmission power classification: nodes with smaller depths (closer to the backbone core) maintain higher transmission power to maintain network connectivity; nodes with larger depths appropriately reduce power to reduce collisions and energy consumption. This hierarchical adjustment process uses a nonlinear decay function, so that the smaller the depth, the slower the rate at which transmission power decreases with increasing depth, thereby minimizing the transmission power of edge nodes while ensuring the stability of the backbone link.

[0035] On the other hand, when the channel occupancy rate is below the second threshold (i.e., the channel is relatively idle) and the instantaneous speed of the node exceeds the speed threshold, the processor determines that the network topology is changing rapidly due to the high-speed movement of the node. At this time, it automatically increases the transmission power of the corresponding node to compensate for the signal attenuation caused by the stretching of the link or the increased obstruction. By incorporating both channel occupancy rate and node mobility into the power control decision factors, this system can prioritize energy saving and congestion reduction in static or low-speed scenarios, and prioritize maintaining connectivity in high-speed scenarios, achieving a dynamic balance between energy efficiency and robustness.

[0036] Furthermore, the nonlinear decay function is expressed as: ; in, The network depth of the node, At maximum transmission power, This is an attenuation coefficient that is dynamically adjusted based on channel occupancy; the higher the channel occupancy, the more attenuation occurs. The larger the value; When the instantaneous velocity of a node exceeds a velocity threshold, Multiply by a factor less than 1.

[0037] The aforementioned nonlinear attenuation function defines the node transmit power. With its network depth The decay pattern of (number of hops). Among them, At maximum transmission power, This is an attenuation coefficient that is dynamically adjusted based on channel occupancy: the higher the channel occupancy, the more attenuation occurs. A larger value results in a sharp decrease in transmit power for nodes at greater depths, thus rapidly alleviating network congestion. This exponentially squared decay function ensures that nodes at smaller depths (such as...) have lower transmit power. =1,2) The transmit power decreases gradually to maintain the stability of the backbone link; while the power of edge nodes with greater depth decreases rapidly to achieve energy saving and interference suppression.

[0038] When the instantaneous velocity of a node exceeds a preset velocity threshold, the system adjusts the attenuation coefficient. Multiply by a factor less than 1, i.e., temporarily reduce. This slows down the rate at which transmit power decreases with increasing depth. This adjustment compensates for link quality fluctuations caused by high-speed node movement and prevents communication interruptions due to excessively rapid power degradation. By mapping channel occupancy and node mobility to the value of the attenuation coefficient α, this invention achieves joint adaptive power control for network congestion and topology dynamics.

[0039] The processor is also configured to perform automatic alarm functions for those who fall behind or remain abnormally stationary. Specifically, each helmet node monitors its relative distance to the nearest teammate node in real time. When this distance exceeds a preset safe distance threshold for N consecutive seconds (e.g., N=10), the processor determines that the wearer may have fallen behind, automatically sends a distress signal to all teammates via the self-organizing network, and flashes a red mark on the transparent face mask display of the node in distress. Simultaneously, the processor also monitors the accelerometer output in the nine-axis inertial measurement unit. If the acceleration value remains close to zero for M seconds (e.g., M=5) (i.e., determined to be stationary), an alarm mechanism is also triggered. The thresholds N and M can be dynamically configured by the command center according to the mission scenario. This function effectively ensures that a soldier can receive timely rescue from teammates when unconscious, injured, or lost, improving team safety.

[0040] In addition to the aforementioned UWB and inertial navigation fusion positioning, this system also supports the expansion of other positioning sources to achieve redundancy and cross-checking. For example, in open areas with GPS signals, the processor can automatically receive data from a GPS or BeiDou module (which connects to a standardized interface in a pluggable manner) and loosely fuse the satellite positioning results with UWB relative positioning: using unscented Kalman filtering (UKF) to correct the global coordinate system with the absolute coordinates of satellite positioning, while utilizing UWB relative ranging to suppress the multipath effect of satellite signals. In environments with absolutely no radio signals (such as deep mines or underwater), the processor can switch to a pure inertial dead reckoning mode, relying on the zero-velocity correction (ZUPT) and gait detection algorithms of the nine-axis IMU to maintain a decimeter-level relative positioning for approximately 5 minutes. The automatic switching between these multiple positioning modes is dynamically decided by the processor based on signal quality indicators (such as UWB signal-to-noise ratio and the number of visible satellites), without user intervention.

[0041] Furthermore, the processor is also configured to: adaptively adjust the lower limit of transmit power based on depth and speed, that is, the minimum transmit power of each node is dynamically calculated according to its current depth and instantaneous speed; nodes with greater depth and slower speed are allowed to be reduced to a smaller power value; nodes with smaller depth or faster speed have their minimum power clamped at a higher level.

[0042] Building upon the aforementioned nonlinear power attenuation, the processor further introduces a depth- and speed-adaptive transmit power lower limit adjustment mechanism. Specifically, the minimum transmit power of each node is no longer a fixed constant, but is dynamically calculated based on its current network depth (hop count) and instantaneous speed: nodes with greater depth and slower speed are allowed to reduce their transmit power to a smaller lower limit, thereby maximizing energy savings when far from the backbone core and relatively stationary; while nodes with smaller depth (e.g., close to the backbone center) or faster speed have their minimum transmit power clamped at a higher level to ensure backbone link connectivity and link stability during high-speed movement. By mapping depth and speed together as decision variables for the power lower limit, this system can achieve fine-grained power boundary control under different network topologies and node movement states, avoiding critical link interruptions or high-speed node disconnections due to excessive power reduction, while further improving the energy efficiency of edge nodes.

[0043] Furthermore, the processor, when performing eye-tracking selective enhancement, is also configured to: construct a gaze-driven dynamic information priority queue, including: The system sorts the position information of all teammates within the field of view of the transparent mask display according to the angle difference with the gaze direction; only teammates whose angle difference is less than the dynamic threshold are highlighted; the dynamic threshold is adaptively adjusted according to the dwell time of the current gaze point and the movement speed of the gaze point, and the longer the dwell time or the slower the movement speed, the smaller the dynamic threshold is; the information in the non-gaze area is rendered with gradient transparency, and the transparency is proportional to the angle difference.

[0044] When performing selective eye-tracking enhancement, the processor constructs a gaze-driven dynamic information priority queue. Specifically, the processor first sorts the position information of all teammates within the field of view of the transparent mask display according to their angular difference relative to the wearer's real-time gaze direction; the smaller the angular difference, the closer the teammate is to the current attention center. Subsequently, only teammates with angular differences less than a dynamic threshold are highlighted and magnified, while other information is downgraded. This dynamic threshold is not a fixed value but is adaptively adjusted based on the continuous dwell time of the current gaze point or the movement speed of the gaze point: the longer the dwell time or the slower the movement speed of the gaze point, the smaller the dynamic threshold value, thereby achieving an attention adaptation effect of "narrowing the display range during prolonged gazing and widening the display range during rapid scanning."

[0045] Meanwhile, the processor uses a gradual transparency rendering method for information in non-focused areas. The transparency is directly proportional to the angle difference; that is, information with a larger angle difference is displayed more faintly, rather than simply toggling or uniformly dimming. This continuous transparency control avoids abrupt disappearance or sudden appearance of information in the field of vision, maintaining the wearer's overall perception of the surrounding environment. Through the above strategy of combining dynamic priority queuing with gradual transparency, this system can prioritize the presentation of situational information most relevant to the current visual attention within a limited display area, significantly reducing cognitive load, while preserving background reference information in non-focused areas, achieving efficient and natural human-computer interaction situational awareness display.

[0046] To further improve the real-time performance and anti-interference capability of gaze prediction, an extended Kalman filter state and observation model was designed when fusing the relative eye coordinates and helmet posture angles. Specifically, the state vector includes four dimensions: eye angular velocity, eye angular acceleration, helmet angular velocity, and helmet angular acceleration. The observation values ​​are the gaze direction vector output by the eye-tracking camera and the posture angle vector output by the nine-axis inertial measurement unit. By establishing a coupled dynamic model of eye movement and head movement, the filter can distinguish the contributions of active eye saccades and passive head turns to the gaze, thus accurately predicting the gaze point location even during rapid head rotation. Experiments show that this fusion method can control the prediction error within 5 milliradians and the lead compensation delay is approximately 30 to 50 milliseconds, significantly outperforming Kalman filters based solely on eye movement models.

[0047] Furthermore, the adaptive adjustment rule for the dynamic threshold is as follows: ; in, For the maximum angular difference, The duration of continuous fixation at the current point of gaze. The velocity of the gaze point movement. The preset sensitivity coefficient; when When it increases Approaching ,when When θ increases th Decrease.

[0048] The adaptive adjustment rule for the above dynamic threshold adopts the following exponential function form: ,in The maximum permissible angle difference is preset (i.e., the upper limit of the angle between the gaze direction and the target direction). The duration of continuous fixation at the current point of gaze. β represents the velocity of the fixation point, and β is a preset sensitivity coefficient. When the fixation point moves rapidly ( When it is relatively large, Approaching zero, the exponent term approaches e0=1, thus... Approaching At this time, the system also highlights teammate information within a large angular difference range, achieving broad information enhancement in the scanning state; conversely, when the gaze point is fixed on a certain point for a long time ( When the value is relatively large, the exponential term decays significantly. The system rapidly narrows its focus, highlighting only targets with minimal angular differences, achieving precise information narrowing in a focused state. By placing the coupling term between dwell time and movement speed within an exponentially decaying power, this rule continuously and smoothly adjusts the dynamic threshold, avoiding abrupt switching caused by a fixed threshold, and ensuring that the information enhancement range matches the wearer's visual attention concentration in real time.

[0049] Furthermore, in the gradient transparency rendering, transparency With angle difference The relationship uses a sigmoid function: ; in, The semi-transparency threshold angle, This is the transition steepness coefficient; at the fan-shaped boundary near the gaze direction, the transparency gradient is gentle, while in the region far from the gaze direction, the transparency rapidly approaches its maximum value.

[0050] At the fan-shaped boundary near the direction of gaze (i.e. near At that time, the gradient of transparency change was relatively gentle, and the displayed information gradually transitioned from bright to semi-transparent, avoiding abrupt visual changes; while in areas far from the direction of gaze ( Much larger The transparency rapidly approaches its maximum value (i.e., almost completely transparent, retaining only slight visibility), thus focusing visual attention on teammate information directly related to the current gaze direction, while still preserving the blurred outline of the background environment. This continuous transparency rendering method based on an S-curve function, compared to traditional on / off hiding or linear transparency changes, better matches the physiological sensitivity of the human eye to sudden changes in brightness, reduces attentional distraction caused by abrupt switching of information display, and maintains an overall perception of the surrounding environment, improving visual comfort and the naturalness of situational awareness during prolonged wear.

[0051] In a typical application scenario of this invention, the system works collaboratively according to the following process: First, after each helmet is powered on, the processor automatically constructs an initial global coordinate system in the absence of GPS by using the relative UWB ranging values ​​between nodes through a graph optimization algorithm. Simultaneously, the self-organizing network module completes power initialization based on depth grading and channel occupancy. After the wearer enters the mission area, the processor fuses UWB and inertial navigation data in real time to output centimeter-level relative positions and displays the teammates' positions in the form of a heatmap on the transparent mask display. When the wearer gazes in a certain direction, the eye-tracking module captures the gaze point, and the processor immediately highlights and amplifies the teammate information in that direction, while enhancing the ambient sound in that direction through bone conduction headphones. If the head rotates rapidly, the processor simultaneously fuses the helmet attitude angle to compensate for prediction delay. When a subnet re-encounters after splitting due to communication interruption, the processor automatically performs coordinate merging based on graph Laplacian initial alignment, RANSAC mismatch removal, and four-point transformation, and ensures global consistency through pose graph optimization. The entire system achieves full-chain intelligence from initialization, positioning, communication, display to human-computer interaction, providing reliable technical support for individual soldier collaboration in extreme environments.

[0052] After each power-on or system reset, the processor automatically executes a Power-On Self-Test (POST) process: sequentially checking whether the UWB module outputs ranging values ​​normally, whether the self-organizing network module can establish connections with other nodes, whether the eye-tracking camera captures the pupil, whether the initial attitude of the nine-axis IMU is stable, and whether the transparent mask display can show the test pattern. If any module malfunctions, the processor displays the specific fault code (e.g., "E01: UWB No Signal") in text form in the upper right corner of the transparent mask and plays a prompt tone through the bone conduction headphones. At the same time, the system log is backed up to the command node in the network via the self-organizing network for background analysis. If a critical module (such as the processor or power supply) experiences an unrecoverable failure, the helmet's physical bulletproof structure remains unaffected, but the processor will proactively cut off power to non-essential peripheral modules, retaining only basic positioning and alarm functions to ensure the wearer's basic safety.

[0053] The embodiments of the present invention aim to protect a modular multimodal situational awareness bulletproof smart helmet system, which has the following effects: 1. By automatically constructing a global coordinate system based on graph optimization under conditions without absolute reference points, adaptively increasing the Kalman filter innovation covariance based on received signal strength, impulse response width, and ranging residual in non-line-of-sight conditions to reduce the fusion weight of unreliable ranging values, and reducing the transmission power according to the node network depth and using random sampling consistency to eliminate mismatched node pairs when they meet again after subnet splitting, this application significantly improves the initial convergence speed and continuous tracking accuracy of multi-node cooperative positioning in complex environments such as indoors, tunnels, and forests without GPS or base station coverage. It effectively suppresses positioning drift caused by non-line-of-sight interference and ensures the connectivity of the backbone link when the network is congested. This solves the core contradiction between positioning robustness and self-organizing network energy efficiency that is difficult to balance in existing technologies. 2. By constructing a gaze-driven dynamic information priority queue, the position information of teammates within the field of view of the transparent mask display is sorted according to the angle difference with the gaze direction, and only teammates whose angle difference is less than the dynamic threshold are highlighted and magnified. At the same time, the information in the non-gaze area is rendered with gradual transparency, and the ambient sound in the gaze direction is simultaneously enhanced through bone conduction headphones. This application realizes the dynamic tracking of the wearer's visual attention by information display and auditory enhancement, avoids the distraction problem caused by information overload or fixed display in traditional helmets, significantly reduces the cognitive load of situational awareness, and solves the technical problem of difficulty in quickly and effectively acquiring multi-source information in high-pressure environments such as battlefields or rescue. 3. By fusing the relative eye coordinates obtained from eye tracking with the helmet attitude angles obtained from a nine-axis inertial measurement unit, the state variables of the Kalman filter simultaneously include both eye angular velocity and helmet angular velocity. Predictive compensation is used to make the highlighted display area lead the actual gaze point. This application eliminates the gaze point prediction lag error caused by rapid head rotation, improving the real-time performance and accuracy of eye-tracking interaction during vigorous movement or rapid head turns. This ensures the synchronization accuracy of attention enhancement display and sound gain, solving the display jitter or response delay problem caused by neglecting helmet attitude coupling in existing eye tracking systems in dynamic helmet scenarios. This further enhances the user experience and operational reliability of the overall situational awareness system.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A modular, multimodal situational awareness bulletproof smart helmet system, characterized in that, include: The helmet is equipped with a multi-source heterogeneous positioning and networking module, an eye-tracking module, a nine-axis inertial measurement unit, and a transparent face mask display. and a processor configured to perform the following operations: Multi-node cooperative positioning and ad hoc network communication include: constructing a global coordinate system using graph optimization algorithms based on the relative ranging values ​​between nodes when there is no absolute reference point; when a UWB signal is received and it is determined to be in a non-line-of-sight state, the non-line-of-sight state is comprehensively determined based on the received signal strength, impulse response width, and ranging residual, and the Kalman filter innovation covariance is adaptively increased to reduce the weight of the ranging value in the fusion; when the network is congested, the transmission power is reduced in stages according to the depth of each node in the network, with nodes at lower depths maintaining higher transmission power to maintain the connectivity of the backbone network; Based on the wearer's real-time gaze direction detected by the eye-tracking module, the information display in the corresponding direction is enhanced on the transparent mask display, and the ambient sound in that direction is simultaneously enhanced; When enhancing the ambient sound, the relative eye coordinates obtained by eye tracking are fused with the helmet attitude angle obtained by the nine-axis inertial measurement unit to compensate for the gaze point prediction error caused by rapid head rotation.

2. The modular multimodal situational awareness bulletproof smart helmet system according to claim 1, characterized in that, In multi-node cooperative localization, the processor is also configured to: when two or more subnets meet again after splitting due to communication interruption, perform coordinate system merging based on graph matching, specifically including: detecting overlapping nodes; constructing a least-squares similarity transformation between the coordinate systems of the two subnets using the relative positional relationship of the overlapping nodes; eliminating mismatched node pairs through a random sampling consensus algorithm; calculating the optimal rotation and translation matrix using the four-point method; and performing merging only when the median of the interior point residuals is less than a preset threshold, otherwise triggering the re-initialization of global graph optimization.

3. The modular multimodal situational awareness bulletproof smart helmet system according to claim 2, characterized in that, The coordinate system merging based on graph matching also includes: initial alignment based on graph Laplacian eigenvectors, that is, before using the random sampling consensus algorithm, extracting the adjacency matrices of overlapping nodes in the two subnets respectively, calculating the second small eigenvector of the graph Laplacian matrix of each adjacency matrix, and determining the preliminary correspondence of overlapping nodes by matching the second small eigenvectors.

4. The modular multimodal situational awareness bulletproof smart helmet system according to claim 2, characterized in that, The processor is also configured to: perform global consistency optimization after the coordinate system is merged, that is, the merged global coordinate system is refined by pose graph optimization, wherein the pose node of each node is connected to the ranging edge of its neighboring nodes; add the transformation matrix estimated by the random sampling consensus algorithm during the merging process as a strong prior factor to the pose graph, and perform real-time optimization using a gradient descent and Gauss-Newton hybrid solver.

5. The modular multimodal situational awareness bulletproof smart helmet system according to claim 1, characterized in that, The processor is configured in congestion management of ad hoc network communication to perform hybrid hierarchical power control based on channel occupancy and node mobility, including: Each node periodically measures the channel busy-idle ratio and calculates its own instantaneous velocity using a nine-axis inertial measurement unit. When the channel occupancy rate exceeds the first threshold, the node network depth is used as the classification criterion. When the channel occupancy rate is lower than the second threshold and the node's instantaneous velocity exceeds the velocity threshold, the transmit power is automatically increased. The depth classification uses a nonlinear attenuation function, so that the transmit power attenuation rate of nodes with smaller depths is lower than that of nodes with larger depths.

6. The modular multimodal situational awareness bulletproof smart helmet system according to claim 5, characterized in that, The nonlinear decay function is expressed as: ； in, The network depth of the node, At maximum transmission power, This is an attenuation coefficient that is dynamically adjusted based on channel occupancy; the higher the channel occupancy, the more attenuation occurs. The larger the value; When the instantaneous velocity of a node exceeds a velocity threshold, Multiply by a factor less than 1.

7. The modular multimodal situational awareness bulletproof smart helmet system according to claim 5, characterized in that, The processor is also configured to: adaptively adjust the lower limit of transmit power based on depth and speed, that is, the minimum transmit power of each node is dynamically calculated according to its current depth and instantaneous speed; nodes with greater depth and slower speed are allowed to reduce to a smaller power value; nodes with smaller depth or faster speed have their minimum power clamped at a higher level.

8. The modular multimodal situational awareness bulletproof smart helmet system according to claim 1, characterized in that, The processor, when performing eye-tracking selective enhancement, is also configured to: construct a gaze-driven dynamic information priority queue, including: The position information of all teammates within the field of view of the transparent mask display is sorted according to the angle difference with the gaze direction; only teammates whose angle difference is less than the dynamic threshold are highlighted; the dynamic threshold is adaptively adjusted according to the dwell time of the current gaze point and the movement speed of the gaze point, the longer the dwell time or the slower the movement speed, the smaller the dynamic threshold; the information in the non-gaze area is rendered with gradient transparency, and the transparency is proportional to the angle difference.

9. The modular multimodal situational awareness bulletproof smart helmet system according to claim 8, characterized in that, The adaptive adjustment rule for the dynamic threshold is as follows: ； in, For the maximum angular difference, The duration of continuous fixation at the current point of gaze. The velocity of the gaze point movement. The preset sensitivity coefficient; when When it increases Approaching ,when When θ increases th Decrease.

10. The modular multimodal situational awareness bulletproof smart helmet system according to claim 8, characterized in that, In the gradient transparency rendering, the transparency α is related to the angle difference. The relationship uses a sigmoid function: ； in, The semi-transparency threshold angle, This is the transition steepness coefficient; at the fan-shaped boundary near the gaze direction, the transparency gradient is gentle, while in the region far from the gaze direction, the transparency rapidly approaches its maximum value.

Images