A multi-modal collaborative interaction and adaptive anti-disturbance method for complex environment and helmet

Abstract

The application provides a multi-modal collaborative interaction and adaptive anti-interference method and helmet for complex environment, wherein the method comprises: acquiring pose environment data; extracting yaw angular velocity, analyzing head rotation direction when a head turning action causing the yaw angular velocity to be greater than a preset threshold occurs, and feeding forward adjusting the spatial pickup weight of a microphone array before noise reduction convergence; when a physical pressing action is detected and the pose environment data represents a state of jolting, calculating a mechanical wave propagation delay based on the distance parameter from the touch component to the microphone array, and generating a transient isolation window; establishing a multi-modal coupled anti-interference logic: when the head turning action causing the yaw angular velocity to be greater than the preset threshold and the physical pressing action occur at the same time, giving the transient isolation window the highest scheduling priority, and performing priority time domain suspension on the speech audio subjected to the spatial pickup weight adjustment within the transient isolation window. The application can cooperatively block sudden wind flow noise and solid sound transmission crosstalk.

Description

Technical Field

[0001] This invention relates to the field of smart wearable devices and human-computer interaction technology, and in particular to a multimodal collaborative interaction and adaptive anti-interference method and helmet for complex environments. Background Technology

[0002] Currently, smart helmets equipped with interactive components such as microphone arrays and physical buttons are widely used in scenarios such as mobile travel and construction work. In actual use, the system typically uses post-processing acoustic algorithms to reduce noise and filter the collected voice audio. This operating mechanism can meet the user's basic communication and device control needs under relatively stable working conditions.

[0003] In complex outdoor applications, wearers are often affected by both variable airflow and physical bumps. When a wearer rapidly turns their head, the angle at which the external airflow impacts the helmet changes instantaneously, generating non-stationary turbulent wind noise on the microphone array surface. Noise reduction models require a certain data convergence period to cope with such non-linear, abrupt airflow changes, making it difficult to avoid interference at the front end in a timely manner. When the road conditions are bumpy and the wearer operates physical buttons, the mechanical vibration generated by the pressing action is transmitted along the helmet body to the pickup device in the form of elastic waves. This structure of solid-borne sound transmission easily overlaps with the user's effective speech in the time-frequency domain.

[0004] In practice, head-turning observation and key input often occur concurrently. At this time, transient wind noise and solid-borne sound crosstalk mix in the same time domain. Existing heterogeneous sensor data processing logic typically employs independent parallel computation mechanisms, rarely establishing underlying time-domain collaborative isolation strategies to address multimodal physical interference. Faced with concurrent data input from multiple sources, the system's underlying bus is prone to scheduling timing conflicts when synchronously performing acoustic filtering and signal alignment, thus limiting the quality of multimodal data capture and processing. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal collaborative interaction and adaptive anti-interference method and helmet for complex environments, in order to solve the technical problems of existing smart helmets that lack underlying temporal collaborative isolation strategies when dealing with concurrent head rotation and button operations, are prone to data scheduling conflicts, and are difficult to effectively avoid transient wind noise and solid-borne sound crosstalk.

[0006] Therefore, the present invention provides the following technical solution:

[0007] A multimodal cooperative interaction and adaptive disturbance rejection method for complex environments is applied to a smart helmet equipped with a microphone array, touch components, accelerometer, gyroscope, magnetometer, barometer, communication module, and speaker, including:

[0008] The system acquires voice audio picked up by the microphone array, physical pressing actions triggered by the touch component, and positional environment data output by the accelerometer, gyroscope, magnetometer, and barometer.

[0009] Extract the yaw rate from the pose environment data. When a head-turning action occurs that causes the yaw rate to be greater than a preset threshold, analyze the head-turning direction. Before noise reduction convergence, adjust the spatial pickup weight of the microphone array based on the head-turning direction.

[0010] When the physical pressing action is detected and the posture environment data indicates that the device is in a bumpy state, the mechanical wave propagation delay is calculated based on the distance parameter from the touch component to the microphone array to generate a transient isolation window.

[0011] Establish multimodal coupling anti-interference logic: When the head-turning action that causes the yaw angular velocity to exceed the preset threshold occurs concurrently with the physical pressing action, the transient isolation window is given the highest scheduling priority. Within the transient isolation window, the voice audio that has been adjusted by the spatial pickup weight is suspended in the priority time domain to collaboratively block sudden airflow noise and solid-borne sound crosstalk.

[0012] Optionally, the microphone array is mounted in a windproof noise-reducing cavity; the method further includes performing acoustic noise reduction processing based on the speech audio, wherein the acoustic noise reduction processing based on the speech audio includes:

[0013] The speech audio that is not suspended in the time domain is input into a pre-trained neural network model for feature separation;

[0014] The acoustic gain coefficient is dynamically adjusted based on the real-time travel speed reflected by the pose environment data.

[0015] The acoustic gain coefficient and the neural network model are used to filter out wind noise and turbulence energy in the speech audio that comes into contact with the microphone array.

[0016] Optionally, the step of processing the pose environment data includes:

[0017] Extract the altitude and air pressure values ​​output by the barometer and the spatial orientation data output by the magnetometer;

[0018] By integrating the altitude and air pressure values, the spatial orientation data, and the spatial attitude angles in the pose and environment data, a positioning calculation is performed to generate an environmental situational awareness result.

[0019] The environmental situation awareness results are uploaded to the cloud server, and navigation prompts are received from the cloud server.

[0020] Optionally, the step of establishing multimodal coupling anti-interference logic further includes:

[0021] Monitor the vertical amplitude output by the accelerometer;

[0022] When the vertical amplitude is greater than the set amplitude condition, the anti-accidental touch time threshold for the physical pressing action is automatically increased;

[0023] When the voice audio is in a continuous input state and no turning action occurs, the voice input channel is given priority network transmission weight, and the frequency of uploading the environmental situation awareness results to the cloud server is reduced simultaneously to ensure communication bandwidth.

[0024] Optionally, after receiving the navigation prompt information sent by the cloud server, the method further includes:

[0025] Real-time assessment of the ambient background noise sound pressure level;

[0026] Based on the ambient background noise sound pressure level and the preset noise compensation curve, the broadcast volume and loudness of the navigation prompt information are adaptively adjusted so that the acoustic penetration of the navigation prompt information is higher than that of the ambient background noise in complex scenarios.

[0027] Optionally, the acoustic noise reduction process further includes:

[0028] Echo cancellation and automatic gain control are performed synchronously on the aforementioned voice audio.

[0029] The echo cancellation process is used to remove acoustic crosstalk caused by the sound played from the speaker, and the noise-reduced target speech signal is output.

[0030] Optionally, adjusting the spatial pickup weights of the microphone array based on the head rotation direction includes:

[0031] The windward and leeward nodes in the microphone array are distinguished based on the direction of head rotation.

[0032] Decrease the signal confidence weight of the windward node, and simultaneously increase the signal confidence weight of the leeward node;

[0033] After the yaw rate returns to a stable range and a set delay period has elapsed, the beamforming matrix is ​​recalculated based on the ambient noise floor and the initial pickup weights are restored.

[0034] Optionally, calculating the mechanical wave propagation delay based on the distance parameter from the touch component to the microphone array includes:

[0035] Capture the initial time point that triggers the physical pressing action, causing a change in the underlying level;

[0036] Based on the material density parameters of the helmet body and the solid acoustic conduction velocity of the smart helmet, the time it takes for mechanical vibration to reach the microphone array along the surface of the helmet body is calculated.

[0037] The time consumption data is used as the propagation delay of the mechanical wave, and combined with the mechanical rebound time span of the touch component and the initial time node, the starting and ending boundaries of the transient isolation window are determined.

[0038] Optionally, the step of performing priority temporal suspension on the speech audio that has undergone spatial pickup weight adjustment within the transient isolation window includes:

[0039] The first-in-first-out buffer queue is used to forcibly intercept the voice audio flowing into the transient isolation window and block the extraction process of audio sampling frames that overlap with the transient isolation window;

[0040] After the transient isolation window ends, the valid voice audio retained before the forced interception takes effect is spliced ​​in the time domain and smoothed and filtered with the voice audio collected normally afterward, and a coherent audio stream with the solid-borne sound crosstalk removed is output.

[0041] This invention also provides a multimodal cooperative interaction and adaptive anti-interference helmet for complex environments, comprising:

[0042] The helmet body, and the central processing unit, memory and communication module integrated inside the helmet body;

[0043] The microphone array, touch control components, accelerometer, gyroscope, magnetometer, barometer, and speaker are all mounted on the helmet body and electrically connected to the central processing unit.

[0044] The memory stores computer program instructions. When the central processing unit calls and executes the computer program instructions, it implements the multimodal collaborative interaction and adaptive disturbance rejection method for complex environments as described in any of the preceding claims.

[0045] The present invention has achieved the following beneficial effects:

[0046] This invention extracts positional environmental data to analyze the head rotation direction and, before noise reduction convergence, adjusts the spatial pickup weights of the microphone array using feedforward adjustment. This allows it to avoid noise interference caused by sudden changes in airflow on the windward side before the noise reduction algorithm converges. By combining the device's turbulence state with the distance parameters from the touch component to the microphone array to calculate the mechanical wave propagation delay, this invention utilizes a generated transient isolation window to accurately define the temporal boundary of solid-borne sound transmission. When faced with concurrent head-turning actions and physical pressing actions that cause yaw angular velocities greater than a preset threshold, the system establishes multimodal coupling anti-interference logic, assigns the transient isolation window the highest scheduling priority, and performs priority temporal suspension of voice audio within the isolation window. This mechanism objectively blocks the synchronous aliasing of sudden airflow noise and solid-borne sound transmission at the underlying data flow level, alleviates computational resource scheduling conflicts during multiple concurrent tasks, and improves the device's pickup continuity and multimodal interaction reliability in complex dynamic environments.

[0047] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0048] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 A system module composition diagram of a smart helmet provided in an embodiment of the present invention;

[0051] Figure 2 A flowchart illustrating a multimodal cooperative interaction and adaptive disturbance rejection method for complex environments, provided by an embodiment of the present invention;

[0052] Figure 3 A flowchart for adjusting the spatial pickup weights of a microphone array using feedforward adjustment, provided in an embodiment of the present invention;

[0053] Figure 4 A flowchart for calculating the propagation delay of mechanical waves to generate a transient isolation window is provided in an embodiment of the present invention;

[0054] Figure 5 This is a flowchart of a process for performing priority temporal suspension on speech audio that has undergone spatial pickup weight adjustment, provided in an embodiment of the present invention.

[0055] Figure 6This is a flowchart of acoustic noise reduction processing based on speech audio provided in an embodiment of the present invention. Detailed Implementation

[0056] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0057] This invention provides a multimodal collaborative interaction and adaptive anti-interference method for complex environments, applied to a smart helmet equipped with a microphone array, touch component, accelerometer, gyroscope, magnetometer, barometer, communication module, and speaker. Furthermore, this invention also provides a multimodal collaborative interaction and adaptive anti-interference helmet for complex environments.

[0058] like Figure 1 As shown, in one embodiment, the smart helmet includes a helmet body, and a central processing unit, a memory, and a communication module integrated inside the helmet body. The microphone array, the touch control component, the accelerometer, the gyroscope, the magnetometer, the barometer, and the speaker are all mounted on the helmet body and electrically connected to the central processing unit. The memory stores computer program instructions; when the central processing unit calls and executes these instructions, it implements the multimodal collaborative interaction and adaptive anti-interference method for complex environments provided in this embodiment of the invention.

[0059] The helmet body is injection molded from high-molecular composite materials such as polycarbonate or carbon fiber, and contains a sealed chamber for mounting electronic components. The microphone array includes multiple spatially distributed microelectromechanical systems (MEMS) pickup nodes, positioned at pre-defined openings on the front, sides, and back of the helmet body. The touch control components are embedded in the external side interaction area of ​​the helmet body, and include microswitches, elastic metal springs, and a hardware filtering circuit composed of capacitors and inductors for de-jittering. The accelerometer, gyroscope, and magnetometer are integrated into an inertial measurement unit (IMU) chip, surface-mounted on the main printed circuit board inside the helmet body near the user's head's geometric center of gravity. The barometer is installed in a hydrostatic chamber inside the helmet body, with its vents covered by a waterproof and breathable damping membrane. The speaker is installed inside the helmet body. The communication module is integrated into the main printed circuit board. The central processing unit is equipped with peripheral interfaces such as the Serial Peripheral Interface (SPI), the Integrated Circuit Bus (I2C), and the Universal Asynchronous Receiver / Transmitter (UART), which establish data transmission channels with each sensor unit through these hardware buses. The memory includes non-volatile flash memory for storing the operating system kernel and algorithm code, and static random access memory (SRAM) for buffering real-time data.

[0060] like Figure 2 As shown, the multimodal collaborative interaction and adaptive anti-interference method for complex environments provided by the embodiments of the present invention includes: acquiring voice audio picked up by the microphone array, physical pressing action triggered by the touch component, and pose environment data output by the accelerometer, the gyroscope, the magnetometer and the barometer.

[0061] Specifically, to address the time-domain deviation caused by inconsistent sampling rates of heterogeneous sensors, the central processing unit (CPU) is equipped with a global hardware timer to provide a system reference clock. Each pickup node of the microphone array senses changes in external airborne sound pressure and converts them into analog electrical signals. The audio analog-to-digital converter (ADC) discretizes and quantizes the analog electrical signals at a set sampling rate (e.g., 16kHz or 48kHz), generating a pulse code modulation (PCM) data stream. The CPU's direct memory access (DMA) controller moves this PCM data stream to a circular buffer in memory and appends a count value of the current system reference clock to the header of each audio data packet, generating a timestamped audio message. When the touch component is closed by external force, the level of its output pin transitions from high to low. The CPU's external interrupt detection unit is configured in falling-edge triggered mode. Upon capturing this level transition edge, the hardware interrupt service routine immediately records the system reference clock parameters at that moment as a timestamp, recording the event as a physical press action. An accelerometer measures linear acceleration along three-dimensional orthogonal coordinate axes, a gyroscope measures the angular velocity components of the carrier's rotation around these axes, a magnetometer measures the three-dimensional spatial magnetic field strength vector, and a barometer measures ambient absolute air pressure using the piezoresistive effect. The central processing unit (CPU) periodically reads the output register values ​​of these sensors according to the operating system's task scheduling cycle. Due to physical differences in the installation orientation of each sensor, the CPU uses a preset coordinate system transformation matrix to project the local carrier coordinate system data output by each sensor onto a global reference coordinate system with the geometric center of the smart helmet as the origin. The integrated spatial coordinates and attitude information are packaged together, and the extracted system clock count is appended to generate pose environment data. Through this timestamp alignment mechanism based on a global clock tree, acoustic level signals, touch trigger signals, and spatial kinematic signals are synchronously registered on a unified absolute time axis.

[0062] The yaw rate in the pose environment data is extracted. When a head-turning action occurs that causes the yaw rate to exceed a preset threshold, the head-turning direction is analyzed. Before noise reduction convergence, the spatial pickup weight of the microphone array is adjusted based on the head-turning direction.

[0063] Specifically, the central processing unit parses the pose environment data sequentially and extracts the data along the vertical gravity axis ( The rotational angular velocity component of the axis is used to construct a discrete sequence of yaw angular velocities. To eliminate the interference of high-frequency mechanical vibrations caused by uneven road surfaces during walking or driving, the central processing unit calls a first- or second-order finite impulse response (FIR) low-pass filter to smooth the yaw angular velocity sequence, configuring its cutoff frequency to filter out frequency bands exceeding the upper limit of normal human head movement frequency. Subsequently, the absolute amplitude of the filtered yaw angular velocity data is compared with a preset threshold stored in memory. This preset threshold represents the angular velocity boundary parameter that distinguishes between minor physiological swaying and conscious turning movements. Based on the measured statistics of neck follow-up during cycling, the yaw angular velocity caused by non-intentional physiological trunk swaying is normally distributed in... to The system preferentially configures the preset threshold within the range as follows: This serves as the lower limit for determining the physical condition of the disturbance. When a head-turning action is determined to cause the absolute amplitude of the yaw rate to exceed the preset threshold, the system state machine marks the current motion state as a head-turning action, extracts the sign bit (positive or negative) of the yaw rate value, and resolves the clockwise or counterclockwise head rotation direction according to the right-hand rule of the coordinate system.

[0064] like Figure 3 As shown, the step of adjusting the spatial pickup weights of the microphone array based on the head rotation direction includes: distinguishing the windward and leeward nodes in the microphone array based on the head rotation direction; lowering the signal confidence weight of the windward node while raising the signal confidence weight of the leeward node; and after the yaw rate returns to a stable range and a set delay period has elapsed, recalculating the beamforming matrix based on the ambient noise floor and restoring the initial pickup weights.

[0065] The central processing unit (CPU) extracts the tangential linear velocity vector corresponding to the head rotation direction and obtains the normal position coordinate vectors of each microphone array pickup node on the helmet surface from the memory. It performs spatial geometric dot product operations to calculate the scouring angle parameters of the relative airflow on each node surface. Based on aerodynamic flow field distribution rules, pickup nodes within the positively compressed area (i.e., the windward section) are identified as windward nodes; pickup nodes located on the back of the helmet or in areas shielded by streamlined structures (i.e., the wake shelter area) are identified as leeward nodes. Subsequently, the CPU intervenes in the underlying beamforming algorithm configuration. In the multiplication and accumulation calculation link of the multi-channel signals, the initial confidence weight of each microphone array pickup node is set to a scalar of 1.0. For the windward nodes, the processor calls the linear attenuation equation. Forced pressure reduction was implemented, among which, The updated confidence weights for the windward node signals. The initial confidence weights for the windward nodes are denoted as . This represents the absolute amplitude of the real-time yaw rate. The preset aerodynamic response constant is set to a value of [value missing]. The single voltage drop threshold is clamped to 0.3 to prevent clipping distortion during front-end analog-to-digital conversion. Based on the principle of global acoustic field gain conservation in the array, the processor extracts the weight difference reduced on the windward side, divides it by the total number of nodes on the leeward side, and uses the resulting quotient as a positive bias, accumulating it one by one into the signal confidence weight of the leeward side node data channel. This zero-sum operation transiently shifts the directional null region of the spatial filter to the high-pressure side. The entire process only calls the microcontroller's Single Instruction Multiple Data (SIMD) multiply-add instructions, avoiding the processing lag introduced by floating-point inversion of the covariance matrix. This proactively blocks nonlinear wind noise signal aliasing from the physical front end during the transient impact of wind turbulence on the windward side nodes.

[0066] After the head-turning action is completed, the pose environment data shows that the yaw rate amplitude drops to a stable range below a preset threshold. Since the boundary layer turbulence on the helmet surface requires a certain amount of time to dissipate and return to a smooth state, the central processing unit starts an internal timer to wait for a set delay period. The value of this set delay period is positively correlated with the curvature of the streamlined mechanical surface of the helmet and the airflow dissipation time. In this embodiment, to ensure that the nonlinear aerodynamic noise generated by the surface boundary layer turbulence is completely dissipated at the physical contact interface, the set delay period is configured to 400 milliseconds. Subsequent environmental noise floor acquisition is performed only after the delay period ends. After the delay period ends, the microphone array is controlled to acquire time-slice data without voice input under the current static posture as an environmental noise floor sequence. This environmental noise floor sequence is used to calculate the spatial covariance matrix of the multi-channel signal. The regular equations of beamforming algorithms such as Minimum Variance Distortionless Response (MVDR) or Delay-and-Sum are then called to resolve for the optimal weight vector that maximizes the gain in the target direction and minimizes the energy in the interference direction. The calculated optimal weight vector is written into the beamforming weight matrix to restore the initial pickup weights.

[0067] When the physical pressing action is detected and the posture environment data indicates that the device is in a bumpy state, the mechanical wave propagation delay is calculated based on the distance parameter from the touch component to the microphone array to generate a transient isolation window.

[0068] Specifically, the central processing unit extracts the pose environment data along the orthogonal axis of gravity ( The system calculates the variance or root mean square extreme value of the linear acceleration parameter sequence of the axis within a set sliding time window. When the variance exceeds the reference constant threshold characterizing the smooth motion state, the system state machine marks the current operating condition as a bumpy state. Specifically, the length of the set sliding time window is configured to be 200 milliseconds. Based on the forced vibration characteristics of the mechanical system caused by normal road conditions, the reference constant threshold characterizing the smooth motion state is preferably set to an acceleration variance of 0.15g² (where g is the gravitational acceleration constant). When the acceleration variance of the pose environment data exceeds this automatic monitoring threshold in two consecutive sliding time windows, the system determines that a bumpy state has been triggered. Under this bumpy state condition, if the external interruption detection unit synchronously captures the physical pressing action triggered by the touch component, the isolation calculation logic for mechanical sound transmission of the structural solid is activated.

[0069] like Figure 4 As shown, the calculation of mechanical wave propagation delay based on the distance parameter from the touch component to the microphone array includes: capturing the initial time node that triggers the physical pressing action and causes a change in the underlying level; calculating the time taken for the mechanical vibration to reach the microphone array along the surface of the helmet body based on the helmet body material density parameter and solid acoustic conduction velocity of the smart helmet; using the time taken as the mechanical wave propagation delay, and combining it with the mechanical rebound time span of the touch component and the initial time node to jointly delineate the start and end boundaries of the transient isolation window.

[0070] The central processing unit extracts the absolute system clock count value that triggered the physical press hardware interrupt as the initial time node. Simultaneously, it extracts the physical constants of the smart helmet's polycarbonate (PC) composite body: material density. With elastic Young's modulus Substituting into the one-dimensional elastic body longitudinal wave group velocity equation The surface stress wave propagation rate was calculated. The speed is approximately 1384 m / s. The geodesic length from the geometric force center of the touch component to each microphone hole in the array is read from the system's 3D digital model mapping file. This distance parameter is preferably constrained to the range of 0.15 meters to 0.25 meters. Division operations yield time data precisely distributed in the range of 0.10 milliseconds to 0.18 milliseconds. This microsecond-level time data is used as the mechanical wave propagation delay, superimposed forward along the initial time node to anchor the initial boundary of the transient isolation window. After the metal spring of the touch component closes under pressure, the aftershock deformation period caused by physical damping is measured and calibrated to be 12 to 15 milliseconds. The system establishes the termination boundary by forcibly superimposing a 15-millisecond constant backward based on the initial boundary. The closed time domain interval jointly defined by the initial and termination boundaries generates the transient isolation window used to control the underlying data flow.

[0071] Establish multimodal coupling anti-interference logic: When the head-turning action that causes the yaw angular velocity to exceed the preset threshold occurs concurrently with the physical pressing action, the transient isolation window is given the highest scheduling priority. Within the transient isolation window, the voice audio that has been adjusted by the spatial pickup weight is suspended in the priority time domain to collaboratively block sudden airflow noise and solid-borne sound crosstalk.

[0072] In practical applications, head-turning actions and physical pressing actions may overlap on the operating system's task scheduling timeline. Under this concurrent state, if weight adjustment and data isolation are performed simultaneously, pointer read / write operations on the underlying audio buffer queue will cause bus resource conflicts. When the CPU's arbitration kernel detects that relevant interrupt flags are simultaneously set, it forcibly assigns the transient isolation window the highest scheduling priority according to a preset multimodal arbitration strategy, suspending preemption requests from other audio processing tasks.

[0073] like Figure 5 As shown, the step of performing priority temporal suspension on the speech audio that has undergone spatial pickup weight adjustment within the transient isolation window includes: forcibly intercepting the speech audio flowing into the transient isolation window using a first-in-first-out buffer queue, and blocking the extraction process of audio sampling frames overlapping with the transient isolation window; after the transient isolation window ends, temporally splicing and smoothing filtering are performed on the valid speech audio retained before the forced interception took effect and the subsequently normally acquired speech audio, and outputting a coherent audio stream that eliminates the solid-borne sound crosstalk.

[0074] Specifically, when the system's main clock count step falls within the time boundary of the transient isolation window, the CPU modifies the transmission enable register mask of the audio DMA channel. Using a first-in-first-out (FIFO) buffer queue, audio sampling frames with timestamps within that time interval are forcibly intercepted and cleared / overwritten in the buffer. Simultaneously, by modifying the software interface's return status bit, the extraction process of audio sampling frames overlapping with the transient isolation window is blocked, achieving time-domain suspension of data. During the time-domain suspension, due to the interruption of continuous valid audio input, the calculation of the covariance matrix will produce singular values. Therefore, the CPU synchronously issues instructions to freeze the adjustment calculations for spatial pickup weights, stop the inversion and iterative updates of the floating-point arithmetic unit in the beamforming module, and maintain the weight parameter state of the previous cycle.

[0075] After the system clock frequency crosses the termination boundary of the transient isolation window, the central processing unit (CPU) releases the transmission restriction of the DMA channel and resumes the normal cyclic acquisition and extraction of voice data frames. For audio timing breakpoints caused by the suspension operation, the CPU retrieves the valid voice audio data retained before the forced interception took effect, as well as the voice audio data acquired normally after the isolation ended. Within a preset overlapping time-domain operation window, a smoothing window function such as the Hanning window or Hamming window is applied to perform scaling weighting and time-domain overlap addition calculations on the amplitude envelopes of the waveform edge data before and after the breakpoint, performing time-domain splicing and smoothing filtering. The initial audio sequence generated by splicing is then input to a low-pass digital filter cascade unit to filter out high-frequency transient phase jumps and pulse artifacts that may be caused by the hard splicing operation at the breakpoint, finally outputting a coherent audio stream free of solid-borne crosstalk. After the time-domain suspension operation ends, the central processing unit immediately re-polulates to extract the latest pose environment data at the current time node, recalculates the yaw rate and relative direction vector, and resumes the beamforming module's adjustment calculation process for spatial sound pickup weights based on the input of the latest pose environment data.

[0076] The microphone array is mounted in a windproof noise-resistant acoustic cavity; the method further includes performing acoustic noise reduction processing based on the speech audio, such as... Figure 6 As shown, the acoustic noise reduction processing based on the speech audio includes: inputting the speech audio that is not suspended in the time domain into a pre-trained neural network model for feature separation; dynamically adjusting the acoustic gain coefficient in combination with the real-time travel speed reflected by the pose environment data; and using the acoustic gain coefficient and the neural network model to filter out the wind noise turbulence energy in the speech audio that contacts the microphone array.

[0077] Specifically, the helmet body has concave airflow channels at the physical mounting positions of the corresponding microphone array nodes, with a waterproof and breathable damping mesh at the front end, forming a windproof and noise-resistant cavity. Through the stagnation effect and boundary layer dissipation of the physical structure, the high-velocity airflow from external impacts is transformed into low-velocity air waves, achieving primary damping and wind resistance. In the digital signal processing stage, the central processing unit extracts the continuous audio stream data after time-domain splicing. The one-dimensional time-domain sequence is windowed and framed according to a preset frame length and inter-frame overlap rate, and the Short-Time Fast Fourier Transform (STFT) operator is used to map the data to the frequency domain, extracting the amplitude spectrum tensor of the complex spectrum. The amplitude spectrum tensor is used as a feature input to a pre-trained neural network model stored in the system's non-volatile memory. This neural network model is pre-trained using a programmable wind tunnel with multi-gradient calibration wind speeds, combined with massive amounts of noisy data collected from human voice samples, and trained and quantized using a backpropagation algorithm. Limited by the static random access memory (SRAM) resources and thermal power consumption constraints of edge microcontrollers in wearable devices, the model abandons the deep stacking architecture and adopts a lightweight topology of a single-layer gated recurrent unit (GRU) cascaded with a single-layer fully connected layer. All synaptic weights and bias parameters are compressed into 8-bit fixed-point integers (INT8) via post-training quantization (PTQ) before deployment, strictly limiting the computational time for single-frame audio frequency domain forward inference to within 2 milliseconds. In real-time forward inference, the neural network model calculates the posterior probabilities of wind noise and speech signals in each frequency sub-band, outputting a frequency domain feature separation mask matrix (such as an ideal proportional masking IRM or a computationally computed masking CBM) with values ​​between 0 and 1.

[0078] Simultaneously, the central processing unit retrieves pose environment data from two consecutive system polling cycles. By calculating the physical distance difference between the three-dimensional positioning coordinates and differentiating the time interval, or integrating the linear acceleration parameters, the absolute translational velocity scalar of the smart helmet relative to the ground is derived, i.e., the real-time travel speed. The velocity-gain correlation mapping logic establishes physical piecewise constraints based on the boundary layer fluid dynamics effects measured in wind tunnels: when the real-time travel speed... At that time, no boundary layer stripping occurred in the front-end wind field, and the system maintained an acoustic gain coefficient of 1.0 in the low-frequency core band from 0 to 600 Hz, achieving full transmission. When boundary layer turbulence intensifies, a linear attenuation voltage drop operator is applied to the 0 to 600 Hz subband. Dynamically generate the frequency domain attenuation multiplier corresponding to the diagonal weight matrix, where, This is the frequency domain attenuation multiplier dynamically generated for this sub-band. For the real-time movement speed of the smart helmet. Once This induces the shedding of vortices from an unsteady KAMAN vortex street, causing broadband masking energy overload. The processor then forcibly clamps the gain of this low-frequency band to 0.2 absolute lower limit. This hard-coupled physical scalar correction mechanism prevents the fixed-point quantization neural network from outputting a distortion mask in the high wind pressure overload frequency band, compensating for the processing limits of edge computing power when dealing with nonlinear turbulence.

[0079] Subsequently, the arithmetic logic unit of the central processing unit performs a Hadamard product operation on the acoustic gain coefficient matrix and the feature separation mask matrix output by the neural network model to generate the final filter mask with comprehensive hydrodynamic parameter correction. The final filter mask is then multiplied element-wise with the complex spectrum matrix of the original speech audio to objectively suppress wind noise and turbulence energy in the frequency domain amplitude dimension. The processed frequency domain matrix is ​​then restored to the time domain signal by inverse short-time fast Fourier transform (ISTFT), outputting the denoised audio sequence.

[0080] The acoustic noise reduction process further includes: performing echo cancellation processing and automatic gain control synchronously on the speech audio; using the echo cancellation processing to remove acoustic crosstalk caused by the sound played by the speaker, and outputting the noise-reduced target speech signal.

[0081] Specifically, the smart helmet has a miniature speaker assembly embedded inside, which generates physical acoustic crosstalk when it emits sound into the internal cavity. The central processing unit (CPU) extracts the downlink baseband digital audio bitstream sent to the speaker's digital-to-analog converter (DAC) at the hardware mixing output control bus and allocates it as the reference excitation signal for the echo cancellation module. The Normalized Minimum Mean Square Error (NLMS) adaptive filter algorithm model is invoked, using the reference excitation signal as input. The output error is calculated, and the internal tap matrix weights of the Finite Impulse Response (FIR) filter are dynamically and iteratively updated according to the adaptive step-size equation to estimate the objective spatial acoustic transfer function between the speaker's emitting point and the microphone's pickup point. Discrete convolution is performed on the reference excitation signal and the estimated acoustic transfer function to generate an estimated transient pseudo-echo digital sequence. In the actual speech time-domain sequence acquired by the microphone array and pre-processed, the pseudo-echo digital sequence is subtracted in reverse, strictly aligned with the clock timestamps, to achieve echo cancellation and remove acoustic crosstalk caused by the speaker's sound playback.

[0082] At the output of the echo cancellation module, the central processing unit initiates the automatic gain control (AGC) calculation process. This control unit continuously calculates the extreme value or root mean square value of the envelope level of the residual speech audio within a specific sliding time window. The calculated level parameters are compared with the system's preset target signal-to-noise ratio (SNR) reference level upper and lower limit threshold range. In this embodiment, the specific sliding time window is 20 milliseconds, and the system's preset target SNR reference level upper and lower limit threshold range is set to -18 dBFS to -3 dBFS. When the extreme value of the envelope level is lower than the lower limit of the reference level range, the gain mapping function is invoked to increase the digital amplification gain multiplier parameter; when the extreme value of the envelope level approaches the peak clipping threshold of the full-scale digital-to-analog conversion, a compression-limiting mechanism is triggered to decrease the gain multiplier parameter. Through dynamic closed-loop level correction compensation, the dynamic range of the audio signal is locked within the set range, outputting a smooth, noise-reduced target speech signal. The gain mapping function uses logarithmic companding mapping logic: when the extreme value of the envelope level... At that time, the gain multiplier parameter Conversely, when When approaching the critical threshold, a hard inflection point with a limiting ratio of 4:1 is triggered to reduce the gain and prevent output peak clipping.

[0083] The steps for processing the pose environment data include: extracting the altitude and pressure values ​​output by the barometer and the spatial orientation data output by the magnetometer; fusing the altitude and pressure values, the spatial orientation data, and the spatial attitude angles in the pose environment data, performing positioning calculations to generate environmental situation awareness results; uploading the environmental situation awareness results to a cloud server, and receiving navigation prompts from the cloud server.

[0084] Specifically, in the absence of external positioning signals, the central processing unit extracts the absolute atmospheric pressure physical quantity sensed and output from the barometer register. It then calls the atmospheric temperature gradient compensation calibration formula to convert the absolute atmospheric pressure physical quantity into a vertical elevation displacement variable relative to the starting point of movement, thereby obtaining the altitude pressure value. Specifically, the atmospheric temperature gradient compensation calibration formula uses the general barometric altimeter conversion equation: .in, This is the vertical elevation displacement variable (i.e., altitude air pressure value) relative to the starting point of the movement. This is the current absolute pressure reading of the barometer. The atmospheric reference pressure of the starting point for movement, initialized when the system starts up. The ambient absolute temperature at the time of initialization. The standard temperature lapse rate is taken as a constant of 0.0065 K / m. This is the universal gas constant. It is the acceleration due to gravity. The air molar mass is used. This classical equation provides the relative vertical elevation to withstand temperature drift. The Hall voltage component data from the triaxial magnetometer is extracted, and after compensation calibration using preset soft and hard magnetic interference calibration matrix parameters, the absolute yaw angle relative to the Earth's magnetic north pole coordinate system, i.e., the spatial azimuth data, is calculated. The soft and hard magnetic interference calibration matrix parameters are generated during the equipment manufacturing process based on the standard ellipsoid fitting algorithm. The corresponding calibration compensation formula is: ,in, To calibrate the compensated three-dimensional magnetic field vector, This is the original three-dimensional magnetic field vector output by the magnetometer. This is the zero-bias translation vector caused by interference from the hard magnetic components inside the helmet. This is a 3×3 orthogonal scaling compensation matrix used to eliminate distortion in soft magnetic materials. and The parameters are stored in non-volatile memory for real-time retrieval.

[0085] At the multi-sensor data fusion algorithm level, the central processing unit (CPU) constructs an Extended Kalman Filter (EKF) mathematical observation model. In the state prediction update step, the system's prior state prediction matrix is ​​updated using spatial attitude angles (Eulerian angles or quaternion form) derived from the pose environment data via continuous integration of gyroscope angular velocity signals and translational coordinate parameters generated by the quadratic integration of linear acceleration. In the measurement update step, altitude and air pressure values ​​are used as independent external measurement parameters in the vertical dimension, and spatial orientation data is used as the heading observation constraint vector in the horizontal two-dimensional plane. By solving the observation residual matrix, the Kalman gain weights and system error covariance matrix are recursively calculated to perform closed-loop correction, offsetting the coordinate physical drift errors caused by inertial sensor bias and temperature drift integration. After the Kalman filter equation converges, a structured data packet composed of spatial three-dimensional position parameters and orientation attitude is output, generating the environmental situation awareness result. The CPU drives the baseband unit of the communication module through a general-purpose serial interface to encapsulate the environmental situation awareness result according to the transmission protocol and initiate upload communication to the corresponding cloud server. The cloud server platform combines the environmental situational awareness messages with the map database to perform calculations and generate navigation prompt packages corresponding to the path and nodes. The smart helmet communication module receives the package in the downlink radio frequency link, parses and executes it in the local decoding layer, generates navigation prompt information, and sends it to the audio buffer for rendering.

[0086] The steps for establishing multimodal coupling anti-interference logic further include: monitoring the vertical amplitude output by the accelerometer; when the vertical amplitude is greater than a set amplitude condition, automatically increasing the anti-mistouch time threshold for the physical pressing action; when the voice audio is in a continuous input state and no turning action occurs, assigning priority network transmission weight to the voice input channel, and simultaneously reducing the frequency of uploading the environmental situation awareness results to the cloud server to ensure communication bandwidth.

[0087] The central processing unit extracts accelerometer data at a 200Hz beat. Axis data, calling an infinite impulse response high-pass filter with a cutoff frequency set to 5Hz for forced stripping. The static gravity component is used to calculate the root mean square of the high-frequency transient differential current within a 50-millisecond sliding time window, outputting a dimensionless vertical amplitude scalar. Under optimal smooth operating conditions, the standard mechanical residual vibration period of the touch component's metal spring is distributed between 5 and 8 milliseconds. The system fixes the interrupt debouncing timer reload reference of the general-purpose input / output (GPIO) peripherals to 12 milliseconds. When the vertical amplitude scalar exceeds the pre-calibrated value... When the boundary threshold is reached, it is confirmed as a strong forced vibration condition, where the contact spring is prone to self-oscillation frequency jumping due to inertial impact. The processor immediately overwrites the hardware timer mask, forcibly pulling the debouncing comparison constant to 35 milliseconds. Within the expanded anti-accidental touch monitoring window, any low-level transition pulse train caused by helmet vibration with a pulse width of less than 35 milliseconds is judged as an invalid mechanical glitch by the underlying state machine and discarded at the hardware level. Only when the timing span of the pin's stable low level exceeds this 35-millisecond dynamic tolerance limit is the system confirmed to report a valid physical pressing action, cutting off illegal consecutive presses at the hardware source.

[0088] Simultaneously, the CPU's built-in voice endpoint detection logic module analyzes the short-time average energy and short-time zero-crossing rate statistics of the audio buffer data in real time. When the analysis confirms that the audio features are in a fully effective state within the time window, it is marked as a continuous input state. At the same time, the CPU retrieves the integral of the yaw rate from the gyroscope. If no significant deflection displacement (i.e., no turning action) is confirmed, the system assesses and confirms that the current core business requirement is continuous audio communication. The operating system kernel task scheduler responds to this logical judgment, modifying the priority description field in the Task Control Block (TCB), forcibly assigning priority network transmission weight to the voice input channel, ensuring priority for audio data processing and RF communication baseband air interface transmission resources. Simultaneously, the CPU modifies the timer control parameters, extending the cycle period for acquiring pose environment data according to a set proportional scaling step size, and simultaneously reducing the frequency of uploading the environmental situational awareness results to the cloud server to ensure communication bandwidth. The proportional scaling step size is set to a constant 1.5. When continuous voice input is confirmed, the system reads the underlying SPI hardware bus clock cycle of the pose environment data, extending it from an initial 10 milliseconds cycle to 15 milliseconds and then 22.5 milliseconds, with the maximum time interval hard-clamped at 50 milliseconds. This mechanism effectively releases backbone bandwidth resources by yielding to the peripheral sampling frequency. By reducing the frequency of background environment data transmission and channel occupancy, it actively releases RF communication bandwidth and processing unit time slices, ensuring the continuity and reliability of the front-end voice input channel resources.

[0089] After receiving the navigation prompt information sent by the cloud server, the method further includes: assessing the ambient background noise sound pressure level in real time; and adaptively adjusting the broadcast volume and loudness of the navigation prompt information in a specific frequency band based on the ambient background noise sound pressure level and a preset noise compensation curve, so that the acoustic penetration of the navigation prompt information in complex scenarios is higher than that of the ambient background noise.

[0090] Specifically, before sending navigation prompts to the downlink digital-to-analog converter amplifier circuit, the central processing unit (CPU) calls a separate bypass calculation thread. This thread uses the voice endpoint detection module to automatically locate the blank segment of discrete-time data containing only ambient noise, which does not contain valid voice input. It performs root-mean-square energy integration on the audio sample frames within this time segment and calls the logarithmic conversion formula to perform physical mapping transformation, quantizing the output as a decibel scalar reflecting the absolute energy density of the current external sound field, i.e., the ambient background noise sound pressure level. A discrete matrix lookup table, i.e., a preset noise compensation curve, is embedded in a non-volatile partition of the memory. This discrete matrix lookup table defines the exact piecewise linear correspondence between the ambient background noise sound pressure level and the global scaling factor: when the sound pressure level... At that time, the scaling factor remains at 1.0; Within the interval, the coefficient increases linearly with a slope of 0.025 / dB; when the sound pressure level At that time, the scaling factor is capped at 1.75. Simultaneously, the local frequency band gain parameter is configured such that when the ambient sound pressure level triggers... At the threshold, a fixed +3dB auditory compensation gain is applied within a specific frequency range of 1000Hz to 3000Hz. The calculated ambient background noise sound pressure level (in decibels) is used as an index parameter and substituted into the curve matrix to find and calculate the global scaling factor and local frequency band gain parameters that match the corresponding ambient noise intensity.

[0091] The central processing unit (CPU) extracts the global scaling factor and drives the digital front-end amplification link to proportionally and synchronously increase the macroscopic digital amplitude reference of the digital envelope of the navigation prompt information, thereby adjusting the broadcast volume. Simultaneously, the CPU allocates the digital multi-band parametric equalizer network components. For specific frequency bands covering the energy distribution area of ​​the main human speech consonant perception formants (typically calibrated to the 1000 Hz to 3000 Hz range), additional quality factor and gain constant extraction instructions are applied to perform additional decibel gain enhancement processing on this range, thereby adjusting the loudness of the specific frequency band. Through the superposition of the above-mentioned global proportional volume enhancement and targeted energy enhancement of specific target frequency bands, the acoustic energy frequency distribution structure of the digital audio output signal is adjusted. This gives the navigation prompt information sound waves ultimately driven by the speaker the acoustic penetration property to resist the masking limitations of external broadband environmental background noise, ensuring that the acoustic penetration of the navigation prompt information in complex scenarios is higher than that of the environmental background noise.

[0092] All the methods and implementation steps described above rely on the physical attribute acquisition mechanism of the smart helmet hardware entity. Combined with time-domain clock alignment operation at the data bus level, arbitration configuration strategy for task scheduling priority, spatial coordinate transformation and filtering solution model, time-frequency feature mapping and adaptive array matrix adjustment formula, a closed-loop interactive control logic system from data input to instruction feedback is constructed.

[0093] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal collaborative interaction and adaptive anti-interference method for complex environments, applied to a smart helmet equipped with a microphone array, touch component, accelerometer, gyroscope, magnetometer, barometer, communication module, and speaker, characterized in that, include: The system acquires voice audio picked up by the microphone array, physical pressing actions triggered by the touch component, and positional environment data output by the accelerometer, gyroscope, magnetometer, and barometer. Extract the yaw rate from the pose environment data. When a head-turning action occurs that causes the yaw rate to be greater than a preset threshold, analyze the head-turning direction. Before noise reduction convergence, adjust the spatial pickup weight of the microphone array based on the head-turning direction. When the physical pressing action is detected and the posture environment data indicates that the device is in a bumpy state, the mechanical wave propagation delay is calculated based on the distance parameter from the touch component to the microphone array to generate a transient isolation window. Establish multimodal coupling anti-interference logic: When the head-turning action that causes the yaw angular velocity to be greater than the preset threshold occurs concurrently with the physical pressing action, the transient isolation window is given the highest scheduling priority. Within the transient isolation window, the voice audio that has been adjusted by the spatial pickup weight is suspended in the priority time domain to collaboratively block the sudden change noise of wind flow and solid-borne sound crosstalk. The step of performing priority temporal suspension on the speech audio that has undergone spatial pickup weight adjustment within the transient isolation window includes: The first-in-first-out buffer queue is used to forcibly intercept the voice audio flowing into the transient isolation window and block the extraction process of audio sampling frames that overlap with the transient isolation window; After the transient isolation window ends, the valid voice audio retained before the forced interception takes effect is spliced ​​in the time domain and smoothed and filtered with the voice audio collected normally afterward, and a coherent audio stream with the solid-borne sound crosstalk removed is output.

2. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 1, characterized in that, The microphone array is installed in a windproof noise-absorbing cavity; The method further includes performing acoustic noise reduction processing based on the speech audio, wherein performing acoustic noise reduction processing based on the speech audio includes: The speech audio that is not suspended in the time domain is input into a pre-trained neural network model for feature separation; The acoustic gain coefficient is dynamically adjusted based on the real-time travel speed reflected by the pose environment data. The acoustic gain coefficient and the neural network model are used to filter out wind noise and turbulence energy in the speech audio that comes into contact with the microphone array.

3. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 1, characterized in that, The steps for processing the pose environment data include: Extract the altitude and air pressure values ​​output by the barometer and the spatial orientation data output by the magnetometer; By integrating the altitude and air pressure values, the spatial orientation data, and the spatial attitude angles in the pose and environment data, a positioning calculation is performed to generate an environmental situational awareness result. The environmental situation awareness results are uploaded to the cloud server, and navigation prompts are received from the cloud server.

4. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 3, characterized in that, The steps for establishing multimodal coupling anti-interference logic also include: Monitor the vertical amplitude output by the accelerometer; When the vertical amplitude is greater than the set amplitude condition, the anti-accidental touch time threshold for the physical pressing action is automatically increased; When the voice audio is in a continuous input state and no turning action occurs, the voice input channel is given priority network transmission weight, and the frequency of uploading the environmental situation awareness results to the cloud server is reduced simultaneously to ensure communication bandwidth.

5. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 3, characterized in that, After receiving the navigation prompt information sent by the cloud server, the method further includes: Real-time assessment of the ambient background noise sound pressure level; Based on the ambient background noise sound pressure level and the preset noise compensation curve, the broadcast volume and loudness of the navigation prompt information are adaptively adjusted so that the acoustic penetration of the navigation prompt information is higher than that of the ambient background noise in complex scenarios.

6. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 2, characterized in that, The acoustic noise reduction process also includes: Echo cancellation and automatic gain control are performed synchronously on the aforementioned voice audio. The echo cancellation process is used to remove acoustic crosstalk caused by the sound played from the speaker, and the noise-reduced target speech signal is output.

7. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 1, characterized in that, The step of adjusting the spatial pickup weights of the microphone array based on the head rotation direction includes: The windward and leeward nodes in the microphone array are distinguished based on the direction of head rotation. Decrease the signal confidence weight of the windward node, and simultaneously increase the signal confidence weight of the leeward node; After the yaw rate returns to a stable range and a set delay period has elapsed, the beamforming matrix is ​​recalculated based on the ambient noise floor and the initial pickup weights are restored.

8. The multimodal cooperative interaction and adaptive disturbance rejection method for complex environments according to claim 1, characterized in that, The calculation of the mechanical wave propagation delay based on the distance parameter from the touch component to the microphone array includes: Capture the initial time point that triggers the physical pressing action, causing a change in the underlying level; Based on the material density parameters of the helmet body and the solid acoustic conduction velocity of the smart helmet, the time it takes for mechanical vibration to reach the microphone array along the surface of the helmet body is calculated. The time consumption data is used as the propagation delay of the mechanical wave, and combined with the mechanical rebound time span of the touch component and the initial time node, the starting and ending boundaries of the transient isolation window are determined.

9. A multimodal collaborative interaction and adaptive disturbance rejection helmet for complex environments, characterized in that, include: The helmet body, and the central processing unit, memory and communication module integrated inside the helmet body; The microphone array, touch control components, accelerometer, gyroscope, magnetometer, barometer, and speaker are all mounted on the helmet body and electrically connected to the central processing unit. The memory stores computer program instructions. When the central processing unit calls and executes the computer program instructions, it implements the multimodal cooperative interaction and adaptive anti-disturbance method for complex environments as described in any one of claims 1 to 8.

Images