Portable AI mental early warning and intervention terminal integrated with multi-modal acquisition

Abstract

The application relates to the cross field of intelligent hardware and mental health technology, and discloses a portable AI mental early warning and intervention terminal integrated with multi-mode collection. The terminal is based on a heterogeneous core main control chip, and a logic architecture containing a bottom layer collection, quality evaluation, compensation control, fusion calculation and inference intervention is constructed. The bottom layer collection module realizes high-precision clock alignment of heterogeneous data through asymmetric core binding and clock callback compensation. The quality evaluation module extracts the confidence of each mode in real time, and adjusts the hardware backlight or performs audio frequency shift masking to improve the original signal quality when the quality is damaged through the compensation control module. The fusion calculation module uses the confidence to perform feature physical level attenuation and splicing. The inference intervention module outputs the state result after time series filtering and smoothing, and drives the execution peripheral to implement hierarchical intervention. The application solves the problems of multi-mode data collection misplacement and signal quality degradation in an uncontrolled environment, and improves the accuracy of terminal early warning and the robustness of closed-loop intervention.

Description

Technical Field

[0001] This invention relates to the intersection of smart hardware and mental health technology, specifically a portable AI-powered mental health early warning and intervention terminal that integrates multimodal data acquisition. Background Technology

[0002] With the development of wearable devices and artificial intelligence technology, using portable terminals to collect users' visual, voice and physiological multimodal data to assess their psychological state and provide intervention feedback has become a technological application trend in the field. However, existing portable psychological state monitoring terminals still face a number of technical limitations when operating in complex physical environments.

[0003] In the multi-source heterogeneous sensor data access stage, due to the different sampling frequencies and hardware communication protocols of visual, audio and physiological sensors, the hardware data stream is often affected by the preemption of bus resources by the general computing tasks of the operating system and the nonlinear delay of the transmission link during the transmission to the main control chip. This causes timestamp jitter and offset when the data of each modality reaches the software processing layer, resulting in misalignment of the heterogeneous modality data acquisition time window, destroying the strict synchronization of multimodal data on the physical time axis, and reducing the spatiotemporal alignment accuracy during feature fusion.

[0004] Meanwhile, portable terminals, when used in uncontrolled physical environments, are affected by external factors such as changes in ambient light, background noise interference, or loose sensor placement, leading to a decline in data quality for some modalities. Existing technologies mostly rely on software algorithms for post-processing filtering or directly discarding low-quality data, lacking an active compensation mechanism for underlying hardware peripherals. When the front-end signal continues to deteriorate, the input features of the fusion inference model are interfered with by noise, causing high-frequency jumps in the terminal's output state prediction results. In addition, when existing terminals detect abnormal states or poor device contact and trigger peripheral feedback, they issue commands with fixed parameters, such as outputting sudden bright screen lights or high-frequency electronic prompts. This abrupt physical feedback can cause startle reflexes or secondary stress in the subject, disrupting the stable environment required for psychological monitoring and making it difficult to build a coherent and secure closed-loop early warning and intervention control link. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a portable AI psychological early warning and intervention terminal that integrates multimodal data acquisition. This solves the problems of poor timing alignment accuracy in heterogeneous data acquisition by existing portable devices, as well as data quality degradation and abrupt closed-loop intervention feedback caused by environmental interference.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a portable AI psychological early warning and intervention terminal integrating multimodal acquisition, the terminal comprising a hardware architecture and a modular architecture, the hardware architecture including a main control chip, a multimodal acquisition front-end, and execution peripherals; the main control chip integrating a low-power core cluster, a high-performance core cluster, and a neural processing unit; the multimodal acquisition front-end including a visual acquisition unit, a voice acquisition unit, and a physiological acquisition unit; the execution peripherals including a touch display peripheral, a speaker peripheral, a status indicator peripheral, and a local solid-state memory; The module architecture, which operates based on the hardware architecture, includes: The underlying acquisition module is used to synchronously acquire multimodal raw data and perform global clock binding; The quality assessment module is used to calculate the quality confidence of each modality and extract physical distance parameters based on time window data; The compensation control module is used to adjust the execution peripherals by combining physical distance parameters and physiological baseline state when the quality confidence level is lower than the threshold. The fusion computing module is used to extract feature vectors from each modality, introduce quality confidence for weighted attenuation and concatenation, and generate a global fusion feature matrix; The inference intervention module is used to load the global fusion feature matrix into the output physical prediction probability of the neural processing unit, and after temporal filtering and smoothing, extract the state category corresponding to the highest probability to drive the execution of the peripheral device call intervention program.

[0007] By adopting the above technical solution, and by using a multi-core heterogeneous architecture main control chip and constructing collaborative logic from the underlying acquisition module to the inference intervention module, a quality assessment mechanism is introduced at the data access end to measure the reliability of each modality of data. When the data quality is damaged, the underlying hardware is pre-triggered to perform compensatory actions to actively improve the front-end signal input conditions. At the same time, the quality confidence of the hardware-level assessment is directly used as the feature attenuation weight in the fusion calculation process, so that the neural processing unit adaptively reduces the calculation weight of inferior features. Finally, the peripheral device is driven through time-series smoothing processing, establishing a closed-loop control link from multi-source asynchronous acquisition, hardware-level quality compensation to physical feedback intervention, thereby improving the operational stability and status warning accuracy of portable terminals in uncontrolled environments.

[0008] Preferably, the specific logic for the underlying acquisition module to perform global clock binding is as follows: An asymmetric core binding operation is performed through the operating system kernel scheduling mechanism to uniformly bind multimodal acquisition front-end data communication tasks to the independent core of the low-power core cluster for operation, and the scheduling priority is upgraded to a real-time first-in-first-out strategy; when the hardware data stream generated by the sensor peripheral reaches the operating system hardware abstraction layer, a system clock capture instruction is triggered to obtain the monotonic timestamp; the monotonic timestamp is compensated using a pre-calibrated static bus transmission delay parameter to back-calculate the physical sampling time; the basic physical data block and the corresponding timestamp are integrated into a unified binary data structure and written into a circular buffer block in memory space in ascending order of timestamp.

[0009] By adopting the above technical solution, asymmetric core binding is implemented at the operating system kernel level, isolating the sensor communication tasks triggered by high-frequency interrupts to run independently on low-power cores. This prevents general computing tasks from preempting the sensor data receiving link. Combined with a real-time first-in-first-out scheduling strategy, the determinism of hardware interrupt response is ensured. Furthermore, the static bus transmission delay parameter is used to reverse correct the timestamp obtained by the software layer, reducing the nonlinear time error introduced by the bus transmission process. High-precision physical clock alignment of heterogeneous modal data is achieved at the underlying level, providing an accurate spatiotemporal reference for subsequent feature fusion.

[0010] Furthermore, the specific logic for calculating the quality confidence of each modality in the quality assessment module is as follows: For the visual channel, the visual quality confidence is jointly calculated based on the calculated effective exposure brightness normalization index and the physical distance parameter extracted from the face bounding box area ratio; for the speech channel, the average frequency domain signal-to-noise ratio is extracted by performing a short-time Fourier transform on the audio data stream, and the speech quality confidence is obtained by scalar multiplying the smoothed mapping transformed signal-to-noise ratio index with the normalized speech activity ratio extracted based on speech activity detection; for the physiological channel, the physiological quality confidence is jointly calculated based on the baseline drift penalty coefficient mapped from the low-frequency trend component of the electrodermal signal and the standard deviation of the time interval between adjacent peaks of the photoplethysmography pulse wave.

[0011] By adopting the above technical solutions, a dedicated physical index calculation mechanism is constructed for the data acquisition characteristics of multimodal front-ends of portable terminals. The visual channel combines brightness and physical distance to quantify imaging conditions; the voice channel integrates frequency domain signal-to-noise ratio and voice activity ratio to reduce background noise interference in silent segments; the physiological channel combines baseline drift and peak interval standard deviation to identify low-frequency trend changes and high-frequency artifact fluctuations caused by sensor loosening. The evaluation logic achieves objective quantification of the usability status of front-end input data with low computational overhead.

[0012] Preferably, the hardware compensation control logic of the compensation control module when the visual quality confidence level is lower than the trigger threshold is as follows: read the physical distance parameter cached in the local register to perform physical feasibility boundary verification; when it is determined that the physical distance parameter is greater than or equal to the preset boundary validity threshold, extract the physiological baseline arousal index of the test subject for status verification; if the physiological baseline arousal index is greater than or equal to the high arousal stress threshold, start the anti-startle reflex visual compensation strategy, drive the pulse width modulation controller to generate a nonlinear gradually increasing duty cycle sequence to control the touch display peripheral to perform facial supplementary lighting and set the maximum safe duty cycle for truncation protection; at the same time, modify the global color transformation matrix in the system graphics rendering pipeline to lock the screen output color temperature of the touch display peripheral to the low color warm light frequency band.

[0013] By adopting the above technical solution, when visual confidence is low due to low light or backlight environments, the physiological baseline arousal level and physical feasibility boundary are used as pre-verification conditions. When the subject is detected to be in a state of high arousal stress, an anti-startle reflex compensation strategy is implemented. Hardware-level supplementary lighting control is performed through a non-linearly increasing pulse width modulation duty cycle and a locked low color warm light frequency band. While ensuring effective exposure of the visual front end, the system avoids secondary stress reflexes caused by transient stimulation of high-brightness cold light, thus ensuring the safety of the system execution in the pre-intervention stage.

[0014] Furthermore, the hardware compensation control logic of the compensation control module when the physiological quality confidence level is lower than the trigger threshold is as follows: read the soothing background audio sequence currently being played by the speaker peripheral, perform a fast Fourier transform to obtain the main energy frequency point, call the frequency shift algorithm to shift and modulate the fundamental frequency of the device adjustment prompt signal to be output to the neighborhood of the main energy frequency point to generate masking audio, apply a smooth envelope function with exponentially increasing characteristics to the masking audio at the digital signal processing level, and finally mix and superimpose the processed masking audio with the soothing background audio sequence to generate output audio to drive the speaker peripheral to emit sound.

[0015] By adopting the above technical solution, when the physiological sensor needs to issue a prompt voice due to a decrease in quality confidence caused by abnormal status, the frequency shift algorithm is used to adjust the fundamental frequency of the prompt voice to be near the main energy frequency band of the ambient background audio to form a masking effect. An exponentially rising envelope is applied to the digital processing layer to achieve a smooth and gradual sound entry, reducing the auditory abruptness of sudden high-frequency electronic prompts, so that the device status prompt voice can be naturally integrated into the background audio sequence, reducing auditory interference to the original state of the test subject.

[0016] Preferably, the specific logic of the fusion computing module extracting the feature vectors output by each independent feature extraction branch is as follows: extracting the spatiotemporal feature vector of facial micro-expression output by the three-dimensional residual network branch for the visual channel; extracting the speech emotion feature vector output by the acoustic branch for the audio channel; extracting the temporal physiological feature vector output by the one-dimensional convolution branch for the physiological channel; performing element-wise scalar multiplication operations on the scalar corresponding to the calculated modal quality confidence with the feature vector of their respective modality to attenuate low-quality features; concatenating and integrating the attenuated feature vectors in the feature channel dimension into a global fusion feature matrix; then inputting the global fusion feature matrix into a multilayer perceptron for nonlinear dimensional transformation, and performing probability distribution mapping through a normalized exponential function layer.

[0017] By adopting the above technical solution, high-dimensional data of each modality is processed by independent feature extraction branches, and the quality confidence scalar calculated at the bottom layer is multiplied element-wise with the corresponding high-dimensional features. Before being fed into the fusion layer, the weight expression path of low-quality data is reduced, and the multilayer perceptron is guided to automatically shift attention to high-quality modalities in the subsequent dimensional transformation stage. This mechanism can achieve adaptive regulation of feature flow without building additional dynamic attention network nodes, thereby reducing the inference computation overhead of the neural processing unit.

[0018] Furthermore, the specific logic of the inference intervention module in performing time-series filtering and smoothing is as follows: an exponential smoothing filter is introduced in the time dimension, and the high-frequency fluctuation noise of the instantaneous prediction probability is suppressed by assigning different attenuation weights to historical state data and current observation data, so as to obtain a stable prediction probability after smoothing.

[0019] By adopting the above technical solution, exponential smoothing filter calculation is introduced in the time dimension. The temporal inertia of historical states is used to adjust the non-stationary jump of the current transient probability output. The operation logic filters out probability fluctuation noise caused by attitude transformation or external transient stimuli, prevents the execution peripheral from being affected by high-frequency fluctuation commands and causing malfunctions or frequent switching of intervention modes, and maintains the robust continuity of terminal status feedback.

[0020] Preferably, the specific logic of the reasoning intervention module driving the physical execution peripheral to call the matching closed-loop intervention program is as follows: based on the smoothed stable prediction probability, when it is determined that the state is in a severe stress state, a deep auditory intervention command is triggered, and a binaural beat audio stream signal in the preset EEG induction frequency band is sent to the underlying audio digital-to-analog converter, and the stable prediction probability of the severe stress state is used as a gain coefficient to dynamically modulate the audio output amplitude; when it is determined that the state is in a mild stress state, a visual relief intervention command is triggered, and the pulse width modulation controller is called across the software layer to drive the backlight of the touch display peripheral to perform a low-frequency breathing flashing action.

[0021] By adopting the above technical solution, an intervention mechanism that connects hierarchical distribution with direct connection to the underlying control is constructed. For severe stress, a binaural beat audio stream signal is output, and the predicted probability is fed back as a hardware gain adjustment coefficient to achieve smooth adaptive adjustment of the intervention intensity. For mild stress, the pulse width modulation controller is invoked to bypass the graphics rendering layer and directly drive the physical backlight to generate rhythmic breathing flashes, thereby achieving rhythmic guidance of the visual channel. The hierarchical feedback action of hardware and software coordination improves the execution response speed of the closed-loop intervention link.

[0022] Furthermore, the processing logic of the inference intervention module when determining that the subject is in a calm state is as follows: triggering dynamic baseline calibration logic, reading the currently collected physiological sequence and calculating the mean as the physiological observation value at the current moment, using a first-order inertial filtering algorithm to fuse and update the physiological observation value with the locally cached global physiological baseline index to achieve slow adaptive tracking of the baseline; after completing the baseline calibration, integrating the timestamp information, the smoothed stable prediction probability array, and the intervention execution log to generate a structured evaluation report, which is then encrypted and written to the local solid-state storage to complete offline archiving.

[0023] By adopting the above technical solution, when the target is detected to be in a calm state, the baseline update logic is triggered. The first-order inertial filtering operation is used to correct the locally stored baseline data, and the fixed growth cycle drift of physiological indicators with the diurnal rhythm and changes in ambient temperature is tracked. The baseline adaptive tracking mechanism improves the accuracy of the verification threshold of the subsequent compensatory control module and reduces the cumulative identification error of the system during long-term operation. At the same time, structured data encryption and offline disking operations are performed to ensure the physical security of system data.

[0024] Preferably, the deep learning model used in the fusion computing module is configured in the pre-training stage to adopt a modality-level random drop-off strategy, which forces the quality confidence of a certain modality to zero according to a preset random drop-off probability, and uses a focus loss function to calculate the prediction error to update the weights in reverse.

[0025] By adopting the above technical solution, modal-level quality loss perturbation is injected during the network training stage. By setting the confidence of a specific modality to zero according to probability to simulate sensor failure conditions in actual applications, the neural network is prompted to learn in the feature fusion parameter space to avoid over-reliance on a single channel. Combined with the focus loss function, the classification gradient for difficult samples is optimized, thereby improving the fault tolerance and prediction robustness of the in-terminal fusion computing model under single-modal failure conditions.

[0026] This invention provides a portable AI-powered psychological early warning and intervention terminal that integrates multimodal data acquisition. It offers the following advantages: 1. This invention implements asymmetric core binding at the operating system kernel level, isolates high-frequency sensor communication tasks to run independently on low-power cores, and combines a time axis callback compensation mechanism based on bus transmission delay parameters to eliminate scheduling delays and time jitter errors caused by preemption of general computing tasks. It achieves high-precision physical clock alignment of heterogeneous multimodal data at the hardware level, solves the time window misalignment problem caused by asynchronous acquisition of multi-source sensors in complex environments, and provides an accurate spatiotemporal reference for subsequent multimodal feature fusion.

[0027] 2. This invention constructs a low-level quality assessment and hardware linkage compensation mechanism based on multimodal physical indicators. When the confidence level of single-modal data is detected to be low, the system uses the physiological baseline state as a pre-verification and actively triggers the low-level peripheral control logic such as anti-startle reflex visual supplementary lighting or frequency domain masking audio prompts. This mechanism enables the terminal to actively improve the physical input conditions when it encounters physical obstruction or loose wearing, and avoids the secondary stress response of the subject caused by sudden cold light or high-frequency prompt sound, thus ensuring the safety of the monitoring and intervention process.

[0028] 3. In the model inference stage, this invention introduces a feature physical-level attenuation and temporal filtering strategy based on quality confidence. By performing element-wise scalar multiplication between the confidence scalar calculated at the bottom layer and the high-dimensional features of each modality, the interference weight of inferior features can be adaptively reduced without the need to build a complex dynamic attention network, thereby reducing the inference computation overhead of the main control chip. At the same time, it combines an exponential smoothing filter to suppress transient probability jumps, preventing the terminal execution peripherals from being frequently switched or malfunctioning due to transient noise interference, and maintaining the consistency and stability of the system intervention command output. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the logical architecture of the present invention; Figure 2 This is a flowchart of the processing method of the present invention; Figure 3 This is a schematic diagram illustrating the principle of the asynchronous acquisition and timing alignment mechanism of the multi-source heterogeneous sensors of the present invention; Figure 4 This is a data flow structure diagram of the independent modal quality confidence assessment module of the present invention; Figure 5 This is a logic flow diagram of the hardware compensation control strategy under the constraints of the present invention. Figure 6 This is the topology diagram of the neural network for multimodal feature dynamic fusion and state evaluation of the present invention; Figure 7 This is a flow diagram of the state feedback and adaptive intervention control logic of the present invention; Figure 8 This is a distribution diagram showing the convergence trend of the temporal state probability filter of the present invention; Figure 9 This is a distribution diagram of the dynamic distribution characteristics of the multimodal confidence weights in this invention.

[0030] Among them, 10 is the main control chip; 20 is the visual acquisition unit; 30 is the voice acquisition unit; 40 is the physiological acquisition unit; 50 is the touch display peripheral; 60 is the speaker peripheral; 70 is the status indicator peripheral; 80 is the local solid-state memory; 101 is the low-power core cluster; 102 is the high-performance core cluster; 103 is the neural processing unit; 201 is the low-level acquisition module; 202 is the quality assessment module; 203 is the compensatory control module; 204 is the fusion computing module; and 205 is the reasoning intervention module. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 This invention provides a portable AI psychological early warning and intervention terminal integrating multimodal acquisition. The hardware architecture includes a main control chip 10. The main control chip 10 integrates a low-power core cluster 101, a high-performance core cluster 102, and a neural processing unit 103. The main control chip 10 is physically connected to a multimodal acquisition front-end and execution peripherals. The multimodal acquisition front-end includes a visual acquisition unit 20, a voice acquisition unit 30, and a physiological acquisition unit 40. The execution peripherals include a touch display peripheral 50, a speaker peripheral 60, a status indicator peripheral 70, and a local solid-state memory 80.

[0033] Based on the hardware architecture, the system includes the following modules: bottom-level acquisition module 201, quality assessment module 202, compensation control module 203, fusion computing module 204, and inference intervention module 205.

[0034] The underlying acquisition module 201 is used to call the underlying driver interface of the main control chip 10 to synchronously acquire multimodal raw data and perform global clock binding; The quality assessment module 202 is used to extract aligned time window data, calculate the quality confidence of each modality, and extract physical distance parameters. The compensation control module 203 is used to issue instructions to adjust the output parameters of the touch display peripheral 50 or the speaker peripheral 60 when the quality confidence level is lower than a preset threshold, in combination with the physical distance parameter and the physiological baseline state. The inference intervention module 205 is used to receive the initial physical prediction probability and perform temporal filtering smoothing, extract the state category corresponding to the highest probability as the final evaluation result, and drive the physical execution peripheral to call the matching closed-loop intervention program accordingly.

[0035] See attached document Figure 2 This invention provides a portable AI-powered psychological early warning and intervention terminal processing method integrating multimodal data acquisition, comprising the following steps: S10, the underlying acquisition module 201 starts the hardware timed acquisition task. Through the asymmetric core binding mechanism, the visual acquisition unit 20, the voice acquisition unit 30 and the physiological acquisition unit 40 are scheduled in parallel on the low-power core cluster 101. The global real-time clock is called to assign an absolute monotonic timestamp to each frame of data stream, and the data structure carrying the monotonic timestamp is written into the circular buffer. S20, the quality assessment module 202 extracts aligned time window data from the circular buffer, independently calculates the confidence scores of visual quality, speech quality, and physiological quality, and simultaneously extracts the proportion of face area in the visual image frame as a physical distance parameter and writes it into the cache. S30, the compensation control module 203 judges the quality confidence of each mode. When any quality confidence is lower than the trigger threshold, it reads the cached physical distance parameter to perform physical feasibility boundary verification. Under the premise that the physical boundary conditions allow and a specific physiological baseline is met, it takes over the underlying hardware driver and outputs a nonlinear pulse width modulation signal to control the touch display peripheral 50 to perform facial ambient lighting, or drives the speaker peripheral 60 to output a mixed adjustment prompt tone through a frequency domain masking mechanism. S40, the fusion computing module 204 extracts the feature vectors output by each independent feature extraction branch, introduces the visual quality confidence, speech quality confidence and physiological quality confidence calculated by the system as dynamic attention weight allocation factors, and performs element-wise scalar multiplication of the quality confidence scalar of each modality with the feature vector of the corresponding modality to achieve physical-level attenuation of low-quality features. S50, the inference intervention module 205 concatenates and integrates the various feature vectors after confidence weighting into a global fusion feature matrix in the feature channel dimension. The global fusion feature matrix is ​​loaded into the neural processing unit 103 to perform nonlinear dimension transformation and probability distribution mapping of the multilayer perceptron in parallel, and outputs physical prediction probabilities containing multiple class states. After temporal filtering and smoothing, the state class corresponding to the highest probability is extracted as the final result. Based on this, the state indication peripheral 70 is driven to perform physical state transformation and the matching intervention program is called on the touch display peripheral 50 and the speaker peripheral 60. All feature data is encrypted and written to the local solid-state storage 80 to complete offline archiving.

[0036] See attached document Figure 3The underlying acquisition module 201 is responsible for acquiring and clocking multi-source heterogeneous data at the underlying level. Considering that multiple sensors may experience interrupt delays and time window misalignments due to sudden changes in the operating system's computational load during concurrent sampling, the underlying acquisition module 201 implements processor resource isolation and global unified timing on the underlying data link. The underlying acquisition module 201 performs asymmetric core binding operations through the operating system kernel scheduling mechanism, statically dividing the processing core resources inside the main control chip 10 into a low-power core cluster 101 and a high-performance core cluster 102 according to the instruction set architecture characteristics.

[0037] The underlying acquisition module 201 categorizes data communication tasks involving the device's physical bus. These tasks include the mobile industrial processor interface reading thread for the vision acquisition unit 20 and the audio bus interrupt response routine for the voice acquisition unit 30. It also covers the internal integrated circuit bus polling task for the physiological acquisition unit 40. The underlying acquisition module 201 calls the process scheduling application programming interface of the embedded operating system to bind the above tasks to the independent core of the low-power core cluster 101. By setting the kernel's scheduling policy parameters, the scheduling priority of the underlying acquisition threads is upgraded to a real-time first-in-first-out strategy. With the help of the hardware resource physical isolation mechanism, the subsequent computationally intensive evaluation and inference tasks are constrained to the high-performance core cluster 102. This avoids high-concurrency computing from competing for the CPU time slice of the acquisition threads, thereby ensuring the relative stability of the basic physical signal sampling frequency.

[0038] The underlying acquisition module 201 uses a hardware global real-time clock to assign timestamps to each frame of data stream. When the hardware data stream generated by the sensor peripheral reaches the hardware abstraction layer of the operating system, a system clock capture instruction is immediately triggered. The capture instruction specifically reads the continuous elapsed time provided by the underlying hardware counter to obtain the monotonic timestamp. Since the high-speed serial interface used by the vision acquisition unit 20 and the low-speed serial bus used by the vision acquisition unit 40 have inherent differences in data packet encapsulation and physical layer transmission rate, if only the system time of data arrival at the hardware abstraction layer is used as the alignment benchmark, it will cause static misalignment of cross-modal data at the moment of physical sampling. Therefore, the underlying acquisition module 201 has preset static delay calibration parameters for each modal hardware transmission link. The parameters are used to perform time axis callback compensation calculation on the obtained monotonic timestamp to back-calculate the real physical sampling time.

[0039] The underlying acquisition module 201 reconstructs a unified feature timing alignment format based on the acquired clock data and pushes the encapsulated heterogeneous data into a circular buffer queue. It defines the continuous sampling sequence number of the data frame and the independent index of the system modal channel. The underlying acquisition module 201 integrates the basic physical data block and the corresponding timestamp into a unified binary data structure in the memory stack. The specific timing encapsulation operation relationship with delay compensation is expressed as follows: ; In the formula, This is the encapsulated binary data structure; This is the frame sequence number during the continuous sampling process, and its value is a positive integer greater than or equal to 1; It is an independent index for the system modal channels, and its value belongs to the set of visual, speech and physiological modalities; For the first The basic physical data block of a frame; A monotonic timestamp representing the instant a frame data stream is captured upon entering the hardware abstraction layer; The static bus transmission delay parameter is pre-calibrated for the corresponding modal channel, and its value range is calibrated from 10 microseconds to 50 milliseconds, depending on the characteristics of the hardware bus protocol.

[0040] The underlying acquisition module 201 writes the binary data structures generated independently by each channel into a circular buffer block located in memory space in ascending order of timestamps. The subsequent algorithm module defines a fixed-duration interception window by comparing the timestamp differences of the binary data structures in different channels, thereby achieving precise alignment and splicing of heterogeneous data frames. The design of the circular buffer in the computer memory addressing with the first and last connected pointer jump structure and the lock-free concurrent read and write operation logic to avoid concurrent conflicts can be configured in accordance with conventional memory management specifications.

[0041] See attached document Figure 4 The quality assessment module 202 is responsible for measuring the validity of the front-end data chain. It extracts a sequence of aligned data frames with a fixed time window length from the circular buffer, performs grayscale processing on the visual image frames to calculate the brightness histogram, and calculates the ratio of the number of pixels in the effective exposure range to the total number of pixels in the image to obtain the brightness normalization index. At the same time, the quality assessment module 202 calls the face detection algorithm to extract the face bounding boxes in the image, calculates the ratio of the pixel area of ​​the face bounding box to the total pixel area of ​​the image, and stores it in the local register as a physical distance parameter.

[0042] When the subject is not included in the current capture time window, preventing the algorithm from extracting a valid face bounding box, the quality assessment module 202 directly assigns the physical distance parameter to zero and forces the visual quality confidence score within this time window to a minimum value of zero to avoid misjudgment of confidence score caused solely by ambient brightness interference. Under the normal state of successfully extracting the face bounding box, the visual quality confidence score is then derived based on the brightness normalization index and the physical distance parameter. The specific visual assessment operation relationship is expressed as follows: ; In the formula, The confidence level for visual quality ranges from 0 to 1. and These are preset visual calculation weight parameters, both of which have values ​​ranging from 0 to 1 and are always equal to 1; This is a brightness normalization index; It is a physical distance parameter extracted from the ratio of the area of ​​the face bounding box to the total area of ​​the image; The standard face area ratio under optimal recognition conditions is defined as 0.1 to 0.4.

[0043] The quality assessment module 202 performs a short-time Fourier transform on the captured audio data stream to analyze the frequency domain energy distribution, separating the background noise energy band from the effective speech energy band to extract the average frequency domain signal-to-noise ratio (SNR). Using a smoothing mapping logic based on the inverse cotangent function, the average frequency domain SNR is converted into an SNR index with a value range between 0 and 1. Combined with the speech activity detection algorithm, the duration of audio frames determined to contain effective speech actions within a time window is statistically analyzed. The duration is divided by the total duration of the time window to obtain the normalized speech activity percentage. The quality assessment module 202 then performs a scalar multiplication operation between the SNR index and the speech activity percentage to obtain the speech quality confidence level. The specific relationship between the SNR index mapping and the speech quality confidence level calculation is expressed as follows: ; ; In the formula, The signal-to-noise ratio (SNR) after mapping and transformation; The arctangent function is used to perform nonlinear compression on the input decibel difference, achieving a smooth mapping of the numerical value from the infinite domain to the [0, 1] restricted domain; The average frequency domain signal-to-noise ratio extracted by the short-time Fourier transform is expressed in decibels. The reference signal-to-noise ratio threshold is set to a range of 10 dB to 30 dB. The confidence level for speech quality ranges from 0 to 1. This represents the percentage of voice-based activities.

[0044] The quality assessment module 202 reads the skin conductance signal data from the physiological channel, extracts the low-frequency trend component of the skin conductance signal to calculate the absolute value of the skin conductance signal drift slope, and uses an exponential decay function to map it to the baseline drift penalty coefficient. Simultaneously, it uses a peak-finding algorithm to extract the peak position sequence from the photoplethysmography (PPG) data and calculates the standard deviation of the time interval between adjacent peaks. To prevent the physiological sensor from detaching and causing a small number of peaks that would invalidate the standard deviation calculation, the quality assessment module 202 triggers a short-circuit judgment and forces the standard deviation to be set to infinity when the number of peaks is less than three. Simultaneously, it forces the absolute value of the aforementioned skin conductance signal drift slope to be set to infinity so that the baseline drift penalty coefficient is zero. Finally, the quality assessment module 202 calculates the physiological quality confidence level based on the baseline drift penalty coefficient and the standard deviation. The specific relationship between the baseline drift penalty and physiological quality calculation is expressed as follows: ; ; In the formula, This is the baseline drift penalty coefficient; The penalty sensitivity coefficient ranges from 1 to 5; This represents the absolute value of the slope of the skin conductance signal drift. The confidence level for physiological quality ranges from 0 to 1. and For physiological calculations, the weight parameters are both in the range of 0 to 1 and their sum is always equal to 1. The standard deviation of the time interval between adjacent peaks; The maximum permissible physiological signal jitter threshold is set to a range of 0.1 seconds to 1.0 seconds.

[0045] For the grayscale histogram statistical process and the details of the face detection algorithm based on the conventional convolutional neural network model involved in the aforementioned processing, as well as the zero-crossing rate determination logic for speech activity detection and the differential threshold determination process for peak finding in the time domain of physiological signals, the underlying software configuration can be referenced from existing mature open-source signal processing frameworks.

[0046] See attached document Figure 5 The compensation control module 203 is responsible for directly calling physical hardware peripherals to intervene in the environment when the underlying signal is damaged, bypassing the software layer. The compensation control module 203 polls and judges the confidence of each modal quality output from the upstream and sets the hardware compensation trigger threshold. The value range of the hardware compensation trigger threshold is set to 0.4 to 0.6. When the compensation control module 203 detects that the visual quality confidence is lower than the trigger threshold, it reads the physical distance parameter cached in the local register to perform physical feasibility boundary verification.

[0047] A preset boundary validity threshold is used as the criterion for judging the physical supplementary lighting limit. The value range of the boundary validity threshold is defined as 0.05 to 0.1. When the compensation control module 203 determines that the physical distance parameter is less than the boundary validity threshold, it indicates that the current distance of the subject from the device screen is too far and exceeds the effective coverage range of the optical physical supplementary lighting. At this time, the compensation control module 203 determines that it is currently in a physical dead zone and triggers a boundary rejection command to cancel the hardware compensation action for the visual channel in order to avoid ineffective energy consumption and light pollution.

[0048] When the compensation control module 203 determines that the physical distance parameter is greater than or equal to the boundary validity threshold, the compensation control module 203 extracts the physiological baseline arousal index of the current subject for state verification. The physiological baseline arousal index is obtained by mapping the normalized difference operation of the average physiological observation at the current moment with the locally cached global physiological baseline index to characterize the real-time activation level of the sympathetic nervous system. A high arousal stress threshold is set as the basis for determining whether the subject is in a state of high tension. The value range of the high arousal stress threshold is set to 0.65 to 0.85. If the physiological baseline arousal index is less than the high arousal stress threshold, it means that the subject is in a normal calm state. The compensation control module 203 calls the conventional algorithm to drive the touch display peripheral 50 to increase the backlight brightness for conventional supplemental lighting. If the physiological baseline arousal index is greater than or equal to the high arousal stress threshold, the compensatory control module 203 activates the anti-startle reflex visual compensation strategy and sends control parameters to the backlight driver at the bottom layer of the main control chip 10.

[0049] The compensatory control module 203 drives the pulse width modulation controller to generate a non-linear, gradually increasing duty cycle sequence to control the touch display peripheral 50 to perform facial illumination and sets a maximum safe duty cycle for cutoff protection. Simultaneously, the compensatory control module 203 modifies the global color transformation matrix in the system's graphics rendering pipeline to lock the screen output color temperature of the touch display peripheral 50 to a preset low color temperature warm light band. The color temperature value of the low color temperature warm light band is set between 2700K and 3200K to avoid visual stimulation caused by high color temperature cold white light. The specific anti-startle duty cycle control calculation relationship is expressed as follows: ; In the formula, for The pulse width modulation duty cycle of the output at any given time; To allow for the maximum safe duty cycle of the output, the maximum safe duty cycle is set to a range of 80% to 95%. To trigger the initial duty cycle; The slope gradient coefficient is set to a range of 0.01 to 0.05. The current system time; The initial moment for compensation triggering.

[0050] When the compensation control module 203 detects that the confidence level of physiological quality is lower than the set physiological trigger threshold, the range of the physiological trigger threshold is set to 0.3 to 0.5, indicating that the physiological acquisition unit 40 has poor contact and needs to output an adjustment prompt command. The compensation control module 203 reads the soothing background audio sequence currently being played by the speaker peripheral 60. If it is detected that the current system is in the silent test phase, resulting in the background audio queue being empty, the compensation control module 203 directly calls the preset standard pink noise waveform in the local non-volatile storage medium as the masking reference, and directly uses it as the time domain signal of the aforementioned soothing background audio sequence into the subsequent audio mixing and superposition operation. At the same time, it directly specifies the preset pink noise center frequency as the main energy frequency. If background audio is present, a Fast Fourier Transform is performed on the soothing background audio sequence to obtain the main energy frequency. The compensation control module 203 uses the main energy frequency as the masking reference for the subsequent prompt audio rate modulation, thereby using the frequency domain masking effect to reduce the abruptness of the prompt sound to the test subject. The specific main energy frequency extraction operation relationship is expressed as follows: ; In the formula, The main energy frequency point to be extracted; The extremum extraction operator for the independent variable represents the search for the target frequency value that maximizes the amplitude of a given function. For Fast Fourier Transform operators; This is the time-domain signal of the currently playing soothing background audio sequence; This represents the current system sampling time.

[0051] The compensation control module 203 calls a frequency shift algorithm to shift and modulate the fundamental frequency of the device adjustment prompt signal to be output to the neighborhood of the main energy frequency point to generate masking audio. In order to eliminate phase abrupt changes and pops at the audio splicing points, the compensation control module 203 applies a smooth envelope function with exponentially increasing characteristics to the masking audio at the digital signal processing level. Finally, the compensation control module 203 mixes and superimposes the processed masking audio with the soothing background audio sequence to generate output audio to drive the speaker peripheral 60 to emit sound. The specific smooth envelope function and audio mixing and superposition operation relationship are expressed as follows: ; ; In the formula: It is a smooth envelope function; The current system time; The start time for triggering the prompt tone; This is the time constant, and its value ranges from 50 milliseconds to 200 milliseconds. This is the final output of the mixed audio signal from the speaker peripheral 60; This is the time-domain signal of the currently playing soothing background audio sequence; This is a masked audio signal after frequency shifting and modulation from the base frequency.

[0052] The specific butterfly operation derivation process of the Fast Fourier Transform involved in the aforementioned implementation process, as well as the resampling and phase vocoder implementation mechanism of the digital audio modulated frequency shift algorithm, can be based on the fundamental theory of digital signal processing for low-level software code writing and register configuration.

[0053] See attached document Figure 6 The fusion computing module 204 is responsible for receiving the heterogeneous data stream and quality confidence after front-end processing and performing deep psychological stress state inference. The fusion computing module 204 internally constructs a deep learning model containing three independent feature extraction branches. For the visual channel, the fusion computing module 204 converts consecutive multi-frame face-aligned images into a four-dimensional spatiotemporal tensor containing the number of time frames, spatial width and height, and color channels. The input is a three-dimensional residual network branch containing three-dimensional convolutional layers and max pooling layers to extract the spatiotemporal feature vector of facial micro-expressions. For the audio channel, the fusion computing module 204 inputs a two-dimensional Mel spectrogram tensor containing time steps and frequency windows generated by short-time Fourier transform into an acoustic branch composed of a two-dimensional convolutional layer and a bidirectional long short-term memory network layer to extract speech emotion feature vectors. For the physiological channel, the fusion computing module 204 inputs a normalized one-dimensional physiological sequence into a one-dimensional convolutional branch to extract temporal physiological feature vectors.

[0054] After obtaining the independent feature vectors output from the three branches, the fusion computing module 204 introduces the visual quality confidence, speech quality confidence, and physiological quality confidence calculated from the upstream process as dynamic attention weight allocation factors. The fusion computing module 204 performs element-wise scalar multiplication of the quality confidence scalar of each modality with the corresponding modality's feature vector to achieve physical-level attenuation of low-quality features. Subsequently, the fusion computing module 204 concatenates the three confidence-weighted feature vectors along the feature channel dimension to integrate them into a global fusion feature matrix. The fusion computing module 204 inputs the global fusion feature matrix into a multilayer perceptron containing two hidden layers for nonlinear dimensionality transformation. The specific relationship between confidence weighting and multilayer perceptron fusion operation is expressed as follows: ; In the formula, This is the global fusion feature matrix output by the multilayer perceptron; To modify the activation function of the linear unit, This is the connection weight matrix of the hidden layers of a multilayer perceptron. The visual quality confidence score calculated in the preceding steps, The spatiotemporal feature vector of facial micro-expressions extracted by the branches of the three-dimensional residual network; For feature splicing operators; The confidence score of speech quality calculated in the preceding steps, The speech emotion feature vector extracted from the acoustic branch; For physiological quality confidence level; The temporal physiological feature vector extracted by the one-dimensional convolutional branch; is the bias vector of the hidden layer of the multilayer perceptron.

[0055] The fusion calculation module 204 inputs the global fusion feature matrix into the normalized exponential function layer for probability distribution mapping, outputting a one-dimensional probability array containing multiple elements. Each element in the one-dimensional probability array corresponds to the physical prediction probability of the subject's current state as calm, mildly stressed, and severely stressed. The fusion calculation module 204 extracts the state category corresponding to the element with the largest value in the one-dimensional probability array as the final psychological stress state assessment output. The specific state classification prediction operation relationship is expressed as follows: ; In the formula, The corresponding output of the neural network model Physical prediction probability of stress-like states; For the natural constant Exponential functions with base 0, natural constant The approximate value is 2.718; For the corresponding number The linear classification layer weight vector of class states; This is the global fusion feature matrix output by the multilayer perceptron; The total number of psychological stress state categories is preset, and the value of the total number of state categories is constant at 3; The category index variable is used for the summation operation. The value range of the category index variable is from 1 to the total number of state categories. For the corresponding number The linear classification layer weight vector of class states.

[0056] To ensure the multimodal neural network model has effective parameters, the backend server pre-collects multimodal samples using a standardized Trier social stress test paradigm for supervised learning training. Sample labels are generated by combining scores from professional psychological assessment scales with expert ratings. To address the dead zone problem of single-modal failure that may be caused by feature splicing, the backend server employs a modality-level random dropout strategy during the model's forward propagation training phase. A random dropout probability within a preset range of 10% to 30% is used to force the quality confidence of a particular modality to zero, thus compelling the neural network to not rely on a single feature. Simultaneously, the backend server uses a focal loss function to calculate the prediction error and an adaptive moment estimation optimization algorithm to calculate the gradient to update the weights in reverse. The specific focal loss function operation relationship is expressed as follows: ; In the formula, The value of the focus loss function; This is the stress state category index variable, and the value range of the stress state category index variable is from 1 to the total number of state categories; The total number of psychological stress state categories is preset, and the value of the total number of state categories is constant at 3; In order to target the The class balance coefficient for class states is set to a value range of 0.25 to 0.75. The corresponding output of the neural network model Physical prediction probability of stress-like states; The difficulty level parameter is set to range from 1.0 to 5.0. For the one-hot encoding of the true label of the sample, when the sample truly belongs to the first... When the class state is active, the one-hot encoding of the real tag is 1; otherwise, the one-hot encoding of the real tag is 0. It is a logarithmic function with the natural constant as its base.

[0057] For the calculation methods of stride and padding of the three-dimensional convolutional layers involved in the aforementioned model construction process, as well as the specific gating algebra formulas of the forget gate and input gate in the bidirectional long short-term memory network, plus the momentum decay rate update process and backpropagation derivative chain rule of the adaptive moment estimation optimization algorithm, the underlying computation graph node configuration can be referenced from mainstream deep learning computing frameworks.

[0058] See attached document Figure 7The inference intervention module 205 is responsible for receiving the real-time psychological stress state assessment results output by the neural network model and performing closed-loop hardware adjustment and baseline update operations accordingly. To address the issue that the instantaneous prediction probability output by the neural network in adjacent time frames can be affected by local data noise, causing jumps and frequent start-ups and shutdowns of downstream hardware, the inference intervention module 205 introduces an exponential smoothing filter in the time dimension for temporal filtering. The filtering principle utilizes the correlation between preceding and following physical time series to suppress high-frequency fluctuation noise. The inference intervention module 205 calculates the smoothed stable prediction probability by assigning different attenuation weights to historical state data and current observation data. The specific temporal smoothing filtering operation relationship is expressed as follows: ; In the formula, The current system time corresponds to the [number]th Stable predicted probability of stress-like states after smoothing; This is the time smoothing coefficient, and its value ranges from 0.1 to 0.3. The corresponding number of the current outputs of the neural network model The original physical prediction probability of stress-like states; The previous time step corresponds to the first Stable predicted probability of stress-like states after smoothing; This is the current system sampling time; This is a stress state category index variable, and the value range of the stress state category index variable is set to 1 to 3.

[0059] The inference intervention module 205 extracts the category label corresponding to the maximum value of the smoothed stable prediction probability and executes a graded intervention strategy. When the inference intervention module 205 determines that the current subject is in a state of severe stress, it triggers a deep auditory intervention command. The inference intervention module 205 sends a binaural beat audio stream signal of a specific frequency to the underlying audio digital-to-analog converter. The principle of the intervention is to input sound waves with a small frequency difference to the left and right ears to form a difference frequency beat in the human brain, thereby using the brainwave induction effect to reduce the subject's arousal. The frequency difference between the left and right ears of the binaural beat audio stream signal is set in the Theta wave brainwave induction frequency band of 4 Hz to 8 Hz. In order to avoid the subject's startle reflex caused by the sudden insertion of a high-volume beat signal, the inference intervention module 205 uses the stable prediction probability of the severe stress state as a gain coefficient to dynamically modulate the audio output amplitude. The specific dynamic modulation operation relationship of the intervention audio output amplitude is expressed as follows: ; In the formula, The actual amplitude of the intervention audio output by the speaker peripheral in the current system time; The preset baseline noise level for the intervention audio is set to a range of 40 dB to 50 dB. The probability of stable severe stress state prediction for the current system after time smoothing; The maximum adjustable audio gain compensation value is set to a range of 10 dB to 25 dB. This represents the current system sampling time.

[0060] When the inference intervention module 205 determines that the subject is in a mild stress state, it triggers a visual relief intervention command. In this state, the subject's emotions are in a slightly fluctuating range and high-intensity auditory stimulation is not suitable. The inference intervention module 205 calls the pulse width modulation controller across the software layer to drive the backlight of the touch display peripheral 50 to perform a low-frequency breathing flashing action. The inference intervention module 205 locks the breathing flashing frequency at a preset slow-wave breathing frequency band of six to eight cycles per minute. Through visual guidance, the subject autonomously performs deep breathing actions at the same frequency to achieve the activation of the parasympathetic nervous system of the autonomic nervous system. While performing visual relief intervention, the inference intervention module 205 maintains the normal playback of the ambient background audio to ensure the continuity of the test environment.

[0061] When the inference intervention module 205 determines that the subject is in a calm state, it indicates that the subject's physiological indicators are in the natural relaxation range. To address the natural drift of physiological indicators caused by temperature drift of physiological sensors or changes in metabolic rate during long-term testing, the inference intervention module 205 triggers dynamic baseline calibration logic. The inference intervention module 205 reads the currently acquired one-dimensional physiological sequence and calculates the mean as the physiological observation value at the current moment. The inference intervention module 205 uses a first-order inertial filtering algorithm to fuse and update the physiological observation value with the locally cached global physiological baseline indicators to achieve slow adaptive tracking of the baseline. The specific dynamic update operation relationship of the global physiological baseline is expressed as follows: ; In the formula: This refers to the global physiological baseline indicators after the current system time update; The baseline learning rate parameter is set to a value between 0.01 and 0.05. This refers to the global physiological baseline metrics cached locally at the previous time step. This represents the average physiological observation extracted from the current system time. This represents the current system sampling time.

[0062] After completing the above dynamic intervention and baseline calibration, the inference intervention module 205 integrates timestamp information, smoothed stable prediction probability array, and intervention execution log to generate a structured evaluation report and writes it to local non-volatile storage medium for persistent storage. For the underlying audio digital-to-analog converter buffer queue configuration instructions and the waveform synthesis mechanism of binaural beat audio stream in the digital domain, as well as the relational database read / write interface call rules and multi-threaded asynchronous write lock mechanism involved in the aforementioned implementation process, you can refer to the general technical documents related to computer operating systems for software coding implementation.

[0063] To aid in understanding the technical solution of this invention, the following is an application example based on real-time monitoring of psychological stress among personnel in high-load dispatching positions.

[0064] The terminal device is deployed in front of the work console of the test personnel. The underlying acquisition module starts the timed acquisition task. Through the asymmetric core binding mechanism, the visual acquisition, voice acquisition and physiological acquisition tasks involving bus communication are statically bound to the low-power core cluster of the main control chip for operation. The hardware global real-time clock assigns an absolute monotonic timestamp to each data stream. After the time axis callback compensation operation of the static delay calibration parameters, the heterogeneous binary data structure is written into the circular buffer in memory in the order of increasing timestamp.

[0065] Within a specific time window monitored by the system, a sudden change in lighting occurred in the test environment. The quality assessment module extracted alignment data from the circular buffer and calculated that the current visual quality confidence level dropped to 0.32. At this point, the set hardware compensation trigger threshold was 0.50, and the system met the compensation trigger condition. The compensation control module read the physical distance parameter cached in the local register, which had a value of 0.14, greater than the set boundary validity threshold of 0.08. The system determined that the person being tested was within the effective coverage area of ​​the physical supplementary lighting.

[0066] Subsequently, the compensation control module extracts the physiological baseline arousal index of the current test subject, and calculates the value to be 0.76. The value is greater than the set high arousal stress threshold of 0.70, indicating that the test subject is in a state of high tension. The compensation control module takes over the underlying hardware driver and starts the anti-startle reflex visual compensation strategy. The pulse width modulation controller of the touch display peripheral receives a non-linear gradually increasing duty cycle sequence, and the backlight brightness is slowly increased to the maximum safe duty cycle of 85%. At the same time, the global color transformation matrix locks the screen output color temperature to the low color warm light band of 2800K.

[0067] The fusion computing module receives the feature vectors output by each independent feature extraction branch. For visual features damaged by sudden changes in light, the system introduces a visual quality confidence score of 0.32, as well as a speech quality confidence score of 0.87 and a physiological quality confidence score of 0.92 calculated at the same time, as dynamic attention weight allocation factors. Through element-wise scalar multiplication, the system attenuates the interference weight of low-quality visual features at the physical level. The feature vectors are concatenated and integrated into a global fusion feature matrix in the feature channel dimension, which is then loaded into the neural processing unit to perform nonlinear dimensional transformation and probability distribution mapping.

[0068] The neural processing unit outputs initial physical prediction probabilities for calm, mild, and severe stress states. After time-series filtering using an exponential smoothing filter, the system extracts the highest stable prediction probability for the severe stress state, with a value of 0.75. Based on this, the inference intervention module extracts the severe stress state category as the final evaluation result, triggering a deep auditory intervention command. The underlying audio digital-to-analog converter generates a binaural beat audio stream signal with a frequency difference of 5 Hz between the left and right ears. The system presets a base noise floor amplitude of 45 dB and a maximum audio gain compensation of 20 dB. Combined with the stable prediction probability of 0.75, dynamic modulation is performed, and the final intervention audio amplitude output by the speaker peripheral is 60 dB.

[0069] The following table provides experimental data related to this application example to verify system performance: Table 1: Performance Comparison Test Data of Multimodal Concurrent Acquisition and Compensatory Intervention System According to Table 1 and Figure 8 , Figure 9 It can be seen that in the actual test scenario where the ambient illuminance fluctuates, the comparison system equipped with uncompensated and confidence attenuation logic is limited by the degradation of the underlying physical signal quality, and the time-series state classification accuracy ranges from 63.82% to 78.15%. Moreover, the frequency of hardware execution start-stop jitter caused by local data noise interference is relatively high, and the continuous physiological baseline stress relief rate is at a low level. The system configuration equipped with complete closed-loop intervention control logic, through confidence calculation of the quality assessment module and physical feasibility boundary verification of the compensation control module, implements physical-level feature attenuation in the fusion computing module and performs temporal filtering smoothing in the inference intervention module, improving the time-series state classification accuracy to the range of 89.45% to 93.61%.

[0070] The core innovation of this invention lies in constructing a cross-level closed-loop feedback mechanism that integrates the underlying hardware driver takeover with the feature fusion of deep neural networks. Experimental results show that by relying on nonlinear pulse width modulation control and frequency domain masking mechanism to perform physical compensation, and by introducing a dynamic attention weight allocation factor to correct the global fusion feature matrix, this technical solution suppresses high-frequency fluctuation noise under the limited conditions of multi-source heterogeneous sensors at the front end, reduces the frequency of frequent start-stop jitter of downstream hardware devices, ensures the consistency of the output parameters of the closed-loop intervention program, and improves the evaluation robustness and intervention effectiveness of the terminal system.

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A portable AI-powered psychological early warning and intervention terminal integrating multimodal data acquisition, characterized in that, The terminal includes a hardware architecture and a module architecture; The hardware architecture includes a main control chip, a multimodal acquisition front-end, and execution peripherals; the main control chip integrates a low-power core cluster, a high-performance core cluster, and a neural processing unit; the multimodal acquisition front-end includes a visual acquisition unit, a voice acquisition unit, and a physiological acquisition unit. The execution peripherals include touch display peripherals, speaker peripherals, status indicator peripherals, and local solid-state memory; The module architecture, which operates based on the hardware architecture, includes: The underlying acquisition module is used to synchronously acquire multimodal raw data and perform global clock binding; The quality assessment module is used to calculate the quality confidence of each modality and extract physical distance parameters based on time window data; The compensation control module is used to adjust the execution peripherals by combining physical distance parameters and physiological baseline state when the quality confidence level is lower than the threshold. The fusion computing module is used to extract feature vectors from each modality, introduce quality confidence for weighted attenuation and concatenation, and generate a global fusion feature matrix; The inference intervention module is used to load the global fusion feature matrix into the output physical prediction probability of the neural processing unit, and after temporal filtering and smoothing, extract the state category corresponding to the highest probability to drive the execution of the peripheral device call intervention program.

2. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The specific logic for global clock binding in the underlying acquisition module is as follows: By performing asymmetric core binding operations through the operating system kernel scheduling mechanism, the multimodal acquisition front-end data communication tasks are uniformly bound to the independent cores of the low-power core cluster for operation, and the scheduling priority is upgraded to a real-time first-in-first-out strategy. When the hardware data stream generated by the sensor peripheral reaches the hardware abstraction layer of the operating system, the system clock capture instruction is triggered to obtain the monotonic timestamp. The monotonic timestamp is then compensated by time axis callback using the pre-calibrated static bus transmission delay parameter to back-calculate the physical sampling time. The basic physical data block and the corresponding timestamp are integrated into a unified binary data structure and written into the circular buffer block of the memory space in ascending order of timestamp.

3. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The specific logic for calculating the quality confidence level of each modality in the quality assessment module is as follows: For the visual channel, the visual quality confidence level is calculated by combining the calculated effective exposure brightness normalization index with the physical distance parameter extracted from the face bounding box area ratio. For the voice channel, a short-time Fourier transform is performed on the audio data stream to extract the average frequency domain signal-to-noise ratio. The signal-to-noise ratio index after smooth mapping transformation is multiplied by a scalar with the normalized voice activity ratio extracted based on voice activity detection to obtain the voice quality confidence. For physiological channels, the confidence level of physiological quality is calculated by combining the baseline drift penalty coefficient mapped from the low-frequency trend component of the skin conductance signal with the standard deviation of the time interval between adjacent peaks of the photoplethysmography pulse wave.

4. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The hardware compensation control logic of the compensation control module when the visual quality confidence level is lower than the trigger threshold is as follows: The physical distance parameter cached in the local register is read to perform physical feasibility boundary verification. When the physical distance parameter is determined to be greater than or equal to the preset boundary validity threshold, the physiological baseline arousal index of the test subject is extracted for state verification. If the physiological baseline arousal index is greater than or equal to the high arousal stress threshold, the anti-startle reflex visual compensation strategy is activated. The pulse width modulation controller is driven to generate a non-linear, gradually increasing duty cycle sequence to control the touch display peripheral to perform facial illumination and set the maximum safe duty cycle for truncation protection. At the same time, the global color transformation matrix in the system graphics rendering pipeline is modified to lock the screen output color temperature of the touch display peripheral to the low color temperature warm light band.

5. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The hardware compensation control logic of the compensation control module when the physiological quality confidence level is lower than the trigger threshold is as follows: The system reads the currently playing soothing background audio sequence from the speaker peripheral and performs a Fast Fourier Transform to obtain the main energy frequency. It then calls a frequency shift algorithm to shift and modulate the fundamental frequency of the device adjustment prompt signal to be output to the neighborhood of the main energy frequency to generate a masking audio. At the digital signal processing level, it applies a smooth envelope function with exponentially increasing characteristics to the masking audio. Finally, it mixes and superimposes the processed masking audio with the soothing background audio sequence to generate the output audio that drives the speaker peripheral to produce sound.

6. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The specific logic for the fusion computing module to extract the feature vectors output from each independent feature extraction branch is as follows: For the visual channel, the spatiotemporal feature vector of facial micro-expression is extracted from the output of the three-dimensional residual network branch; for the audio channel, the speech emotion feature vector is extracted from the output of the acoustic branch; for the physiological channel, the temporal physiological feature vector is extracted from the output of the one-dimensional convolution branch. The scalar corresponding to the calculated quality confidence of each modality is multiplied element-wise with the feature vector of its corresponding modality to attenuate low-quality features. The attenuated feature vectors are concatenated and integrated into a global fusion feature matrix along the feature channel dimension. The global fusion feature matrix is ​​input into a multilayer perceptron for nonlinear dimension transformation and then mapped to a probability distribution through a normalized exponential function layer.

7. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The specific logic for the inference intervention module to perform time-series filtering and smoothing is as follows: An exponential smoothing filter is introduced in the time dimension. By assigning different attenuation weights to historical state data and current observation data, the high-frequency fluctuation noise of instantaneous prediction probability is suppressed, and a stable prediction probability after smoothing is obtained.

8. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The specific logic of the inference intervention module driving the physical execution peripheral to call the matching closed-loop intervention program is as follows: Based on the smoothed stable prediction probability, when it is determined that the person is in a state of severe stress, a deep auditory intervention command is triggered, which sends a binaural beat audio stream signal in the preset EEG induction frequency band to the underlying audio digital-to-analog converter, and uses the stable prediction probability of the severe stress state as a gain coefficient to dynamically modulate the audio output amplitude. When a mild stress state is detected, a visual relief intervention command is triggered, which crosses the software layer to call the pulse width modulation controller to drive the backlight of the touch display peripheral to perform a low-frequency breathing flashing action.

9. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The reasoning intervention module's processing logic when determining that the test subject is in a calm state is as follows: The dynamic baseline calibration logic is triggered, the currently acquired physiological sequence is read and the mean is calculated as the physiological observation value at the current moment. The first-order inertial filtering algorithm is used to fuse and update the physiological observation value with the locally cached global physiological baseline index to achieve slow adaptive tracking of the baseline. After the baseline calibration is completed, the timestamp information, the smoothed stable prediction probability array and the intervention execution log are integrated to generate a structured evaluation report, which is encrypted and written to the local solid-state storage to complete the offline archiving.

10. The portable AI psychological early warning and intervention terminal integrating multimodal acquisition according to claim 1, characterized in that, The deep learning model used in the fusion computing module is configured in the pre-training stage to adopt a modality-level random drop-off strategy, which forces the quality confidence of a certain modality to zero according to a preset random drop-off probability, and uses the focus loss function to calculate the prediction error to update the weights in reverse.

Images