A multi-modal emotional signal synchronous acquisition and preprocessing method

Abstract

This invention relates to the field of multimodal emotion computing and intelligent signal processing, specifically to a method for synchronous acquisition and preprocessing of multimodal emotion signals. The method includes: acquiring multimodal heterogeneous data streams, acquiring environmental noise frequency band characteristic parameters and physical timestamps; performing adaptive denoising and baseline calibration on low-frequency physiological signals, retaining target physiological low-frequency fluctuation data; using high-frequency behavioral signal mutation points as the starting point of the target event, searching for physiological response peaks and calculating dynamic delay time differences; elastically shifting the target physiological low-frequency fluctuation data along the time axis based on the physical timestamp and dynamic delay time difference to generate semantic-temporal dual-axis synchronous mapping features; constructing an elastic sliding window for hierarchical slicing, generating a semantically aligned multimodal feature matrix, inputting it into a neural network model to output emotion state classification results, and dynamically updating the preset time window length parameter to form a data processing closed loop. This invention can reduce misjudgments caused by cross-modal mismatches.

Description

Technical Field

[0001] This invention relates to the field of multimodal emotion computing and intelligent signal processing, specifically to a method for the synchronous acquisition and preprocessing of multimodal emotion signals. Background Technology

[0002] With the development of intelligent monitoring, vehicle-mounted sensing and physiological computing technologies, emotion state recognition based on multimodal signals has become an important research direction in driving safety monitoring and medical auxiliary monitoring. In order to achieve accurate perception of changes in people's emotions, the synchronous acquisition and preprocessing analysis of multi-source heterogeneous signals has become particularly important. In multimodal emotion recognition, behavioral signals such as facial expressions and speech typically have high sampling frequencies and high rates of change, while physiological signals such as electrocardiograms, electrodermatology, and electroencephalograms (EEGs) are characterized by delayed responses and low-frequency changes. Current technologies often directly splice data from different modalities according to a unified physical clock or align them using fixed delay parameters. This easily overlooks factors such as the inertia of human physiological responses, environmental noise interference, and device baseline drift, leading to semantic misalignment between behavioral events and physiological responses, affecting the accuracy of subsequent feature extraction and emotion classification results. These problems are particularly pronounced in scenarios involving long-term vehicle driving, complex road vibrations, electromagnetic interference, or unstable sensor contact. Therefore, how to effectively process the collected multimodal heterogeneous data streams, accurately establish the dynamic correspondence between high-frequency behavioral signals and low-frequency physiological signals, and achieve cross-modal semantic alignment on the basis of removing environmental noise and baseline drift, so as to generate multimodal feature data suitable for subsequent model recognition, is of great significance for improving the stability and reliability of emotion recognition results. Summary of the Invention

[0003] The purpose of this invention is to provide a method for the synchronous acquisition and preprocessing of multimodal emotional signals, solving the following technical problems: It avoids the semantic misalignment and cross-modal mismatch caused by relying solely on absolute physical time alignment or using fixed empirical delay values ​​to splice data. It can also take into account the real correspondence between human biological response inertia in extracting behavioral signals and physiological reactions, and realize the structured representation of the complete emotional process, thus providing a more stable input basis for emotional state classification.

[0004] The objective of this invention can be achieved through the following technical solutions: A method for synchronous acquisition and preprocessing of multimodal emotion signals, comprising: Acquire multimodal heterogeneous data streams, extract high-frequency behavioral signals and low-frequency physiological signals from the multimodal heterogeneous data streams, and obtain the frequency band characteristic parameters of environmental noise accompanying the input of the multimodal heterogeneous data streams, as well as the physical timestamps of the high-frequency behavioral signals and low-frequency physiological signals; Based on the characteristic parameters of environmental noise frequency band, adaptive denoising and baseline calibration are performed on low-frequency physiological signals to separate environmental noise and equipment baseline drift data, while retaining the target physiological low-frequency fluctuation data in the low-frequency physiological signals. The mutation point in the high-frequency behavioral signal is extracted as the starting point of the target event. The mutation point is the data point where the time derivative of the high-frequency behavioral signal exceeds the preset mutation threshold. Based on the preset initial length parameter, a preset time window with the starting point of the target event as the starting boundary is constructed, and the initial length parameter is used as the length parameter of the preset time window. The preset time window with the starting point of the target event as the starting boundary is constructed, and the response peak of the target physiological low-frequency fluctuation data is searched within the preset time window. The dynamic delay time difference between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data is calculated. Based on the physical timestamp and dynamic delay time difference, the target physiological low-frequency fluctuation data is elastically shifted forward on the time axis. The time series alignment algorithm is used to perform semantic alignment between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, based on the event occurrence state, to generate semantic time dual-axis synchronous mapping features. An elastic sliding window is constructed based on semantic-temporal dual-axis synchronous mapping features. The elastic sliding window is used to perform layered slicing of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic forward shift of the time axis, generating an independent semantically aligned multimodal feature matrix. The semantically aligned multimodal feature matrix is ​​input into a preset neural network model to output the sentiment state classification result. The length parameter of the preset time window is dynamically updated based on the sentiment state classification result, forming a data processing closed loop.

[0005] Furthermore, based on the characteristic parameters of the environmental noise frequency band, adaptive denoising and baseline calibration are performed on the low-frequency physiological signals to separate the environmental noise and equipment baseline drift data, retaining the target physiological low-frequency fluctuation data in the low-frequency physiological signals, including: Low-frequency physiological signals are decomposed using ensemble empirical mode decomposition algorithm to obtain multi-scale intrinsic mode function components; Based on the frequency characteristics of environmental noise frequency band characteristic parameters and multi-scale intrinsic mode function components, high-frequency intrinsic mode function components of corresponding environmental noise are identified and eliminated. Identify and remove the extremely low-frequency intrinsic mode function components of the corresponding device baseline drift data; The remaining intrinsic mode function components are reconstructed to generate target physiological low-frequency fluctuation data, which includes normal physiological low-frequency fluctuation characteristics caused by biological response inertia.

[0006] Furthermore, the abrupt change point in the high-frequency behavioral signal is extracted as the starting point of the target event. Within a preset time window with the starting point of the target event as the initial boundary, the response peak of the target physiological low-frequency fluctuation data is searched. The dynamic delay time difference between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data is calculated, including: Calculate the time derivative of the high-frequency behavioral signal and mark data points whose time derivative exceeds a preset mutation threshold as the starting point of the target event. A preset time window is constructed by taking the starting point of the target event as the initial boundary and combining it with the length parameter of the preset time window. The cross-correlation function is used to calculate the cross-correlation sequence between high-frequency behavioral signals and low-frequency physiological fluctuations of the target within a preset time window; Extract the relative time offset corresponding to the maximum peak value in the cross-correlation coefficient sequence, and use the relative time offset as the dynamic delay time difference.

[0007] Furthermore, the calculation logic for the cross-correlation coefficients in the cross-correlation coefficient sequence is as follows: Multiply the high-frequency behavioral signals corresponding to each time step within the preset time window with the target physiological low-frequency fluctuation data after being shifted by the relative time offset, sum the products over all time steps, and then divide by the length of the preset time window to obtain the cross-correlation coefficient. The preset time window is determined by the starting point of the target event and the length of the preset time window.

[0008] Furthermore, based on the physical timestamp and dynamic delay time difference, the target physiological low-frequency fluctuation data is elastically shifted forward on the time axis. A time series alignment algorithm is then used to semantically align the high-frequency behavioral signals with the elastically shifted target physiological low-frequency fluctuation data based on the event occurrence state, generating semantic-temporal dual-axis synchronous mapping features, including: Based on the physical timestamp, the dynamic delay time difference is subtracted from the overall time axis of the target physiological low-frequency fluctuation data to generate the target physiological low-frequency fluctuation data after the time axis is flexibly forwarded. The feature vectors of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic shift of the time axis are extracted in the same time domain dimension and used as the common dimension feature vector. Calculate the Euclidean distance between the feature vectors of the common dimension, and construct the distance matrix between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward; The dynamic time warping algorithm is used as a time series alignment algorithm to search for the path with the minimum cumulative distance in the distance matrix; Based on the minimum cumulative distance path, the high-frequency behavioral signals and the target physiological low-frequency fluctuation data after the time axis is elastically shifted are copied and stretched in time step to generate semantic-temporal dual-axis synchronous mapping features aligned in the semantic dimension of the target event occurrence.

[0009] Furthermore, when using the dynamic time warping algorithm to search for the path with the minimum cumulative distance in the distance matrix, the logic for calculating the cumulative distance is as follows: The cumulative distance of the current time step node in the distance matrix is ​​equal to the Euclidean distance between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, and the minimum of the following three values: the adjacent cumulative distance along the time axis of the high-frequency behavioral signal, the adjacent cumulative distance along the time axis of the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, and the adjacent cumulative distance in the diagonal direction.

[0010] Furthermore, an elastic sliding window is constructed based on the semantic-temporal dual-axis synchronous mapping features. This window is then used to perform layered slicing of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic time-axis shift, generating independent semantically aligned multimodal feature matrices, including: Extract the signal energy envelope from the semantic-temporal dual-axis synchronous mapping features; Identify the energy rise start and energy decay end point in the signal energy envelope; The starting point of energy rise and the ending point of energy decay are used as the dynamic boundaries of the elastic sliding window; By utilizing dynamic boundaries, cross-modal data is extracted from semantic-temporal dual-axis synchronous mapping features to generate independent semantically aligned multimodal feature matrices corresponding to the starting point of the target event.

[0011] Furthermore, the semantically aligned multimodal feature matrix is ​​input into a pre-defined neural network model to output sentiment state classification results. Based on these sentiment state classification results, the length parameter of the aforementioned pre-defined time window is dynamically updated, forming a data processing closed loop including: Preset classification confidence threshold; Input the semantically aligned multimodal feature matrix into a pre-defined neural network model, and output the sentiment state classification result and the corresponding classification confidence score; If the classification confidence score is not less than the classification confidence threshold, the sentiment state classification result is deemed valid, the sentiment state classification result is output, and the length parameter of the preset time window is updated using the sentiment state classification result. If the classification confidence score is less than the classification confidence threshold, the sentiment state classification result is deemed invalid, triggering an anomaly marking process. The length parameter of the preset time window is increased as the search range for recalculation until the classification confidence score is not less than the classification confidence threshold, or the number of recalculations reaches the preset maximum number of iterations. If the number of recalculations reaches the preset maximum number of iterations and the classification confidence score is still less than the classification confidence threshold, the current semantically aligned multimodal feature matrix is ​​discarded, and an anomaly mark indicating that the sentiment state cannot be identified is output.

[0012] Furthermore, the multimodal heterogeneous data stream is a data stream collected from medical monitoring equipment or driver monitoring systems; High-frequency behavioral signals include facial micro-expression video frame data and speech audio stream data; Low-frequency physiological signals include electrocardiogram (ECG) signals, skin conductance signals, and electroencephalogram (EEG) signals.

[0013] The beneficial effects of this invention are: 1. This invention performs adaptive denoising and baseline calibration on low-frequency physiological signals based on environmental noise frequency band characteristic parameters, which can effectively separate environmental noise and equipment baseline drift data; while removing external vibration interference and electrode drift, it accurately retains the real physiological emotional fluctuations caused by biological response inertia, providing a high signal-to-noise ratio data foundation for emotional feature extraction. 2. Extracting high-frequency behavioral signal mutation points as event starting points and searching for physiological response peaks within a preset time window to calculate dynamic delay time differences; this method avoids mismatches caused by splicing uniform physical clocks or fixed empirical values, and can dynamically quantify the time delay of behavior and physiological response according to real events, accurately establishing the real correspondence between heterogeneous modalities; 3. This invention uses dynamic delay to elastically shift the time axis of physiological data and uses a time series alignment algorithm for semantic alignment. This mechanism transforms physical alignment into semantic synchronization that takes into account the inertia of biological response, eliminates local misalignment caused by differences in sampling rates and durations of different modalities, and generates dual-axis synchronization features that take into account both physical backtracking and semantic progress. 4. This invention utilizes dual-axis synchronous mapping features to extract the signal energy envelope and automatically constructs an elastic sliding window for layered slicing. Compared with traditional fixed-length slicing windows, this method can dynamically extract cross-modal data according to the true duration of emotional events, avoiding errors caused by irrelevant information redundancy or the truncation of long-term physiological responses. 5. After inputting the feature matrix into the neural network, the system dynamically updates the length parameter of the aforementioned search time window based on the output emotional state classification confidence score, forming a data processing closed loop. This mechanism enables the preprocessing parameters to be adaptively adjusted according to the classification effect, effectively adapting to individual physiological response differences and the migration of working conditions. Attached Figure Description

[0014] The invention will now be further described with reference to the accompanying drawings.

[0015] Figure 1 This is a flowchart illustrating a method for synchronous acquisition and preprocessing of multimodal emotion signals as provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 A method for synchronous acquisition and preprocessing of multimodal emotional signals includes: acquiring multimodal heterogeneous data streams, extracting high-frequency behavioral signals and low-frequency physiological signals from the multimodal heterogeneous data streams, and acquiring the frequency band characteristic parameters of environmental noise accompanying the input of the multimodal heterogeneous data streams, as well as the physical timestamps of the high-frequency behavioral signals and low-frequency physiological signals. Based on the characteristic parameters of environmental noise frequency band, adaptive denoising and baseline calibration are performed on low-frequency physiological signals to separate environmental noise and equipment baseline drift data, while retaining the target physiological low-frequency fluctuation data in the low-frequency physiological signals. The mutation point in the high-frequency behavioral signal is extracted as the starting point of the target event. The mutation point is the data point where the time derivative of the high-frequency behavioral signal exceeds the preset mutation threshold. Based on the preset initial length parameter, a preset time window with the starting point of the target event as the starting boundary is constructed, and the initial length parameter is used as the length parameter of the preset time window. The preset time window with the starting point of the target event as the starting boundary is constructed, and the response peak of the target physiological low-frequency fluctuation data is searched within the preset time window. The dynamic delay time difference between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data is calculated. Based on the physical timestamp and dynamic delay time difference, the target physiological low-frequency fluctuation data is elastically shifted forward on the time axis. The time series alignment algorithm is used to perform semantic alignment between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, based on the event occurrence state, to generate semantic time dual-axis synchronous mapping features. An elastic sliding window is constructed based on semantic-temporal dual-axis synchronous mapping features. The elastic sliding window is used to perform layered slicing of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic forward shift of the time axis, generating an independent semantically aligned multimodal feature matrix. The semantically aligned multimodal feature matrix is ​​input into a preset neural network model to output the sentiment state classification result. The length parameter of the preset time window is dynamically updated based on the sentiment state classification result, forming a data processing closed loop.

[0018] This embodiment provides a synchronous acquisition and preprocessing mechanism for multimodal emotional signals. Specifically, the mechanism is deployed in the driver monitoring terminal and the vehicle edge computing unit of a hazardous chemical transport vehicle. It is used to extract the real correspondence between the driver's facial micro-expressions, voice changes and physiological reactions under conditions of long-term driving, complex road conditions and coexistence of in-vehicle vibration and noise, so as to avoid the emotional semantic misalignment caused by splicing data only according to the physical clock. Specifically, the acquisition side receives multimodal heterogeneous data streams, where high-frequency behavioral signals may include facial video frame sequences and speech audio streams, and low-frequency physiological signals may include electrocardiograms, skin conductance, or electroencephalograms; the system simultaneously records the physical timestamps of each modality and receives environmental noise frequency band characteristic parameters; The environmental noise frequency band characteristic parameters can be obtained jointly by the vehicle-mounted inertial sensor, the microphone noise floor estimation module and the equipment self-test module, such as the vehicle going over speed bumps, electromagnetic interference from the intercom in the car, and loose electrode contact. In a simplified example, suppose that during a certain time period, the video side records facial feature values ​​of 0.2, 0.8, and 0.3 at 0.10 seconds, 0.14 seconds, and 0.18 seconds, respectively, the audio side records tone fluctuation values ​​of 0.1, 0.7, and 0.2 during the same time period, and the skin conductance side records values ​​of 0.30, 0.32, and 0.55 at even lower frequencies; If the data is directly merged at the absolute moment of 0.18 seconds, the sudden frown in the video and the skin conductance before the peak is reached will be regarded as the same state, thus weakening the characteristic representation of real emotional events. Therefore, this embodiment does not directly synchronize and fuse data from the same physical moment. Instead, it first removes environmental noise and baseline drift from the low-frequency physiological channel, retains the target low-frequency fluctuations with biological response inertia, and then uses the behavioral signal mutation point as the event anchor point to search for the peak value of the physiological response to obtain the dynamic delay time difference. For example, if the facial mutation occurs at 1.0 second and the corresponding peak value of the skin conductance occurs at 3.2 seconds, then the dynamic delay time difference for this round is 2.2 seconds. Based on their respective physical timestamps and the dynamic delay time difference, the system flexibly shifts the time axis of low-frequency physiological data forward, so that it no longer only represents the data collected at the 3.2-second mark, but is mapped to data that is semantically related to the behavioral event at the 1.0-second mark; After obtaining the forward-shifted sequence, a time series alignment algorithm is used to align the high-frequency behavioral sequence with the forward-shifted physiological sequence in the event semantic dimension, forming a semantic-temporal dual-axis synchronous mapping feature. Here, one axis retains the physical time reference, which is convenient for system backtracking and auditing; the other axis reflects the semantic process of the event, which is convenient for subsequent classification models to analyze the cross-modal evolution of the same emotional fluctuation. After the synchronization mapping is formed, the system does not use a fixed-length window, but instead constructs an elastic sliding window according to the mapping result. Furthermore, if the sudden change in expression lasts for 0.6 seconds in a certain event, while the physiological recovery takes 4 seconds, the left boundary of the window can be close to the starting point of the sudden change in expression, and the right boundary can be extended to the end point of physiological decay, thereby generating a multimodal feature matrix that covers the entire emotional process. The matrix is ​​input into the neural network and outputs the emotional state classification result, such as a certain category in tension, irritability and calmness, and gives the classification confidence. If a certain type of event is repeatedly judged as valid and highly confident, the system can update the subsequent search window length to a value that better fits the individual driver's reaction characteristics, thus achieving closed-loop adaptation. Furthermore, if video frames are missing during a certain period, but speech and electrodermatology are still effective, the system allows the speech mutation point to be used as the behavior anchor point to continue running; if a large area of ​​physiological signals is lost and no usable peaks can be formed, then only a single-modal anomaly marker is generated during that period and it does not participate in the closed-loop parameter update. If the environmental noise frequency band characteristic parameters indicate the presence of strong vibrations and continuous electromagnetic interference, the system can temporarily suspend the update of the current window length to prevent noise samples from affecting the stability of subsequent parameters. After the hazardous materials transport vehicle had been driving continuously for 6 hours, the driver entered a congested section of the city ring road. Multiple vehicles were merging ahead, and the driver briefly frowned, tightened his jaw, and spoke faster. The camera detected the sudden change in facial expression at 12:16:08.420, the microphone detected an increase in voice intensity at 12:16:08.510, and the electrodermal response (EDR) showed a significant peak at 12:16:10.900. Based on this, the system calculated a dynamic delay of approximately 2.4 seconds and mapped the forward shift of the EDR peak to this behavioral event. After semantic alignment and flexible sliding window, a set of independent feature matrices are formed. The classification model outputs a high-stress and agitation result, and updates the driver's default search window from 3 seconds to 4 seconds to adapt to the driver's long latency autonomic nervous response in congested driving. The purpose of this step is to transform the traditional multimodal preprocessing that relies solely on physical time alignment into semantic synchronous preprocessing that takes into account the inertia of human biological responses, thereby achieving a more stable foundation for emotion recognition input and reducing misjudgments caused by cross-modal mismatches. To further clarify, in order to avoid ambiguity caused by different names for the same technical object throughout the text, in this embodiment, high-frequency behavior signal, high-frequency behavior sequence, and behavior-side data all refer to the same type of behavior channel data object unless otherwise stated, and the difference is only in the expression at different processing stages; The data obtained after denoising and baseline calibration of low-frequency physiological signals are uniformly corresponding to the target physiological low-frequency fluctuation data. After the time axis is elastically shifted forward based on the dynamic delay time difference, this data can be simply referred to as the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward. The above names do not introduce new data categories, but only indicate the name changes of the same physiological data object in different processing stages. In addition, the preset time window length parameter, search window length, and default search window length all refer to the same window length parameter used to search for the peak of physiological response around the starting point of the target event. If abbreviations appear in the following text, they shall be subject to the aforementioned unified definition.

[0019] In a preferred embodiment of the present invention, adaptive denoising and baseline calibration of low-frequency physiological signals based on environmental noise frequency band characteristic parameters, separating environmental noise and equipment baseline drift data, and retaining target physiological low-frequency fluctuation data in the low-frequency physiological signals includes: decomposing the low-frequency physiological signals using an ensemble empirical mode decomposition algorithm to obtain multi-scale intrinsic mode function components; identifying and removing high-frequency intrinsic mode function components corresponding to environmental noise based on the frequency characteristics of environmental noise frequency band characteristic parameters and multi-scale intrinsic mode function components; identifying and removing extremely low-frequency intrinsic mode function components corresponding to equipment baseline drift data; and reconstructing the remaining intrinsic mode function components to generate target physiological low-frequency fluctuation data, wherein the target physiological low-frequency fluctuation data includes normal physiological low-frequency fluctuation characteristics caused by biological response inertia.

[0020] This embodiment provides an adaptive denoising and baseline calibration step for low-frequency physiological signals. Specifically, in the aforementioned vehicle monitoring scenario, relying solely on conventional high-pass or low-pass filtering can easily filter out the slow physiological fluctuations that substantially represent changes in emotional state. Especially when the driver is in a state of continuous tension or agitation, the slow changes in skin conductance and heart rate variability are themselves effective information. Therefore, this embodiment introduces ensemble empirical mode decomposition to decompose the physiological signals at multiple scales and then selectively reconstruct them according to labels. Specifically, after the system decomposes a certain low-frequency physiological sequence, it can obtain several intrinsic mode function components and a residual trend term. For ease of explanation, it is assumed that a segment of electrodermal signal is decomposed into four components: the first component exhibits rapid jitter, and the numerical sequence can be approximated as 0.02, -0.03, 0.01, and -0.02. The second component exhibits low-to-mid-frequency fluctuations, with numerical sequences of 0.05, 0.08, 0.12, and 0.10; the third component shows a smoother response increase, with numerical sequences of 0.10, 0.15, 0.20, and 0.18; the fourth component and the residual term show a slow downward trend, with numerical sequences of 0.40, 0.38, 0.36, and 0.34. If the environmental noise frequency band characteristic parameters indicate that the vehicle is passing through a damaged road surface, the first component can usually be identified as a motion artifact. If the device self-test label indicates that the electrode contact impedance continues to rise, the fourth component and the residual term are more likely to be due to the device baseline drift. At this time, the system removes the first component, the fourth component and the residual term, and only retains the second and third components for reconstruction to obtain the target physiological low-frequency fluctuation data. The processing method aims to distinguish between non-target low-frequency interference and effective low-frequency fluctuations, rather than filtering out all low-frequency components. Non-target low-frequency interference includes device baseline drift, while effective low-frequency fluctuations are physiological response characteristics caused by autonomic nervous system and biological response inertia. The reconstructed sequences can be represented by 0.15, 0.23, 0.32, and 0.28, which better reflect the rise and fall of physiological response after an emotional event compared to the original sequences. Furthermore, to avoid the instability of the processing logic when the label is missing, which would affect the subsequent dynamic delay estimation, the system can degenerate into an automatic identification mode based on the component frequency band statistical characteristics, instantaneous energy and amplitude stability when the characteristic parameters of the environmental noise frequency band are missing. However, its output is still used to determine whether a certain component meets the elimination condition, rather than changing the basic logic of eliminating and retaining the remaining components after identification and reconstruction. Only when the confidence level of a component being identified as an environmental noise component or a device baseline drift component reaches a preset threshold is it removed; components that do not reach the threshold are retained and marked with a low confidence level for reference by subsequent modules; this does not change the processing path of this embodiment, which is mainly to remove noise components, drift components and reconstruct the remaining components, and can reduce the risk of accidentally deleting real physiological fluctuations when the labels are incomplete. As a mechanism for handling abnormal situations, if the characteristic parameters of the ambient noise frequency band are missing, the system can degenerate into an automatic identification mode based on the statistical characteristics of the component frequency band and the amplitude stability, and decide whether to remove the corresponding component according to the aforementioned confidence threshold rule; if the number of components after decomposition is lower than the preset number threshold, it indicates that the signal is stable or the sampling quality is insufficient, and the system can skip the decomposition step and directly use the baseline estimated in the previous stable period for calibration. If the reconstructed signal amplitude is close to zero and lasts for a long time, it will trigger an electrode detachment or sensor failure flag, and this segment of data will not be sent to the subsequent delay calculation module. When the same hazardous chemical transport vehicle enters a continuous curved road section, the high-frequency correction action of the steering wheel causes deformation of the driver's arm and chest electrodes with an amplitude lower than the preset physical interference threshold. The original skin conductance curve shows high-frequency transient noise fluctuations. At the same time, the start and stop of the air conditioning compressor causes the reference potential of the acquisition module to drift slowly. Based on the labels of road vibration enhancement and equipment impedance rise, the system removes the high-frequency transient interference components and extremely low-frequency drift components, retaining only the mid-to-low frequency response components, thereby obtaining a target physiological low-frequency fluctuation curve that can still reflect the driver's continuous stress state. The purpose of this step is to preserve as much of the real emotional physiological inertia information as possible while removing external interference and equipment drift, so as to achieve a reliable input for subsequent dynamic delay estimation.

[0021] In a preferred embodiment of the present invention, extracting abrupt change points in high-frequency behavioral signals as the starting point of a target event, searching for the response peak of target physiological low-frequency fluctuation data within a preset time window with the starting point of the target event as the initial boundary, and calculating the dynamic delay time difference between the high-frequency behavioral signals and the target physiological low-frequency fluctuation data includes: calculating the time derivative of the high-frequency behavioral signals, marking data points whose time derivatives exceed a preset abrupt change threshold as the starting point of the target event; and constructing a preset time window with the starting point of the target event as the initial boundary and in combination with the length parameter of the preset time window. The cross-correlation function is used to calculate the cross-correlation coefficient sequence between high-frequency behavioral signals and low-frequency physiological fluctuation data of the target within a preset time window; the relative time offset corresponding to the maximum peak value in the cross-correlation coefficient sequence is extracted, and the relative time offset is used as the dynamic delay time difference.

[0022] This embodiment provides a calculation step for the dynamic delay time difference between modalities; specifically, after only denoising, although the physiological signal is relatively clean, it is still impossible to answer which frown and which skin conductance rise belong to the same emotional event. If a fixed delay value is still used, for example, uniformly assuming that the behavioral reaction leads the physiological reaction by 2 seconds, the probability of mismatch will be significantly increased when different states such as driver tension, anger and fatigue are mixed. Therefore, this embodiment estimates the dynamic delay one event at a time by detecting abrupt changes in behavioral signals and searching for cross-correlation within the window. Specifically, the system first calculates the time derivative of the high-frequency behavioral signal; taking facial tension features as an example, if the continuous sampling values ​​are 0.20, 0.22, 0.25, 0.70, and 0.74, then the adjacent differences can be approximated as 0.02, 0.03, 0.45, and 0.04. When the preset mutation threshold is 0.30, the position corresponding to the 4th sampling point can be marked as the starting point of the target event; if combined with the preset maximum physiological delay constant of 4 seconds, the system will build a 4-second search window from this starting point. Within this window, the system performs cross-correlation operations on the behavioral signal subsequence and the target physiological low-frequency fluctuation data subsequence, respectively; for ease of understanding, the behavioral event template can be assumed to be... The physiological response fragment can take three candidate groups under different offsets: the one-second offset is... When the offset is 2 seconds When the offset is 3 seconds ; The similarity is low at an offset of 1 second, highest at an offset of 2 seconds, and decreases again at an offset of 3 seconds. The system thus obtains a cross-correlation coefficient sequence, such as 0.35, 0.92, and 0.41. The relative time offset corresponding to the maximum peak is 2 seconds, so the dynamic delay time difference is determined to be 2 seconds. An event-by-event window search mechanism is adopted to adapt to the non-constant physiological response delay characteristics of the same driver under different working conditions; for example, the stress response triggered by a sudden braking emergency is usually faster, while the irritation response caused by slow congestion accumulation may be slower; in this way, the most likely physiological correspondence can be found for each behavioral change. As a mechanism for handling abnormal situations, if multiple behavioral mutation points occur within a window with time intervals less than a preset event interval threshold (e.g., two consecutive frowning and one voice pitch increase within 0.5 seconds), the system can first combine them into a composite event starting point and then perform a single correlation search to avoid repeatedly matching the same physiological peak. If the overall cross-correlation coefficient sequence is low, and the highest value is still below the correlation threshold, it indicates that no reliable physiological corresponding event was found within the window. In this case, the behavioral event can be marked as a transient action rather than a valid emotional event. If the peak value of the physiological signal falls outside the right boundary of the window, the window length can be appropriately expanded and recalculated in subsequent closed-loop steps. After the aforementioned vehicle entered the congested section, the driver frowned noticeably at 12:16:08.420 due to the sudden cutting in front of the vehicle. The time derivative exceeded the threshold, and the system set this moment as the start of the event and searched for the skin conductance response within the next 4 seconds. The search results showed that the cross-correlation value was highest at an offset of 2.4 seconds, so 2.4 seconds was taken as the dynamic delay of this event. In this way, the irritation caused by the vehicle cutting in front and the skin conductance response peak that appeared 2.4 seconds later were bound to the same emotional event. The purpose of this step is to dynamically quantify the time delay difference between behavioral and physiological responses based on events, thereby avoiding cross-modal mismatches caused by fixed empirical values.

[0023] In a preferred embodiment of the present invention, the calculation logic of the cross-correlation coefficient in the cross-correlation coefficient sequence is as follows: multiply the high-frequency behavioral signal corresponding to each time step included in the preset time window with the target physiological low-frequency fluctuation data after being translated by the relative time offset, sum the products under all time steps, and then divide by the length of the preset time window to obtain the cross-correlation coefficient; wherein, the preset time window is determined by the starting point of the target event and the length of the preset time window.

[0024] This embodiment provides a specific method for calculating the cross-correlation coefficient. Specifically, to avoid the problem that the calculation results of the delay time difference cannot be reproduced due to different normalization standards or window lengths used by different modules, this embodiment further limits the calculation logic of the correlation coefficient within the window to ensure the consistency of the dynamic delay acquisition path. Specifically, given the event start point and window length, the system extracts a segment of data within the window from the high-frequency behavior sequence and shifts the target physiological low-frequency fluctuation data according to the candidate offset; multiplies the data one by one according to the time step within the window, sums the products, and divides by the window length to obtain the cross-correlation coefficient under that offset. Using a simplified example, if the window length is 3 time steps, the behavior signal is: The physiological signal, after shifting by 1, becomes Then, by multiplying them step by step, we get 2, 1, 0, summing them up to 3, and then dividing by 3, we get the cross-correlation number 1. If the shift is 2 and then translated downwards, it becomes Multiplying these results in 4, 1, and 0, summing them to 5, and dividing by 3 gives approximately 1.67. This shows that the correlation is higher at offset 2. Although longer sequences and finer-grained sampling are used in actual systems, the underlying computation logic can be executed in the manner described above. Furthermore, to ensure that the calculation logic is consistent with different deployment terminals, the system can first perform unified dimension processing on the data of each time step in the window before entering the product summation. For example, the facial tension, voice intensity and skin conductance amplitude values ​​can be transformed to the same preset value range before performing multiplication and averaging. The dimensional processing here only applies to the numerical scale of the input data and does not change the main logic of cross-correlation calculation, which involves multiplying, summing, and dividing by the preset time window length step by step. As a result, the edge end and the back-end verification end can obtain verifiable and consistent comparison results even when faced with different modalities of the original dimensions. This clear calculation method helps to maintain consistent results between the vehicle edge and the back-end verification end; the edge end is responsible for quickly estimating latency, and the back-end can recalculate using the same formula during post-event spot checks, thereby ensuring data consistency in accident tracing or risk auditing. Furthermore, if the translated physiological sequence is not long enough at the edge of the window, the system can adopt either one of two preset strategies: truncation followed by zero padding or shortening the value segment and then padding to the preset window length. However, the same strategy should be used consistently in the same device to ensure that the denominator always remains the preset time window length. If the window length is zero or there are no effective time steps due to abnormal configuration, the relevant calculations in this round will be directly determined to be invalid and will fall back to the default delay parameter. If data corresponding to certain time steps is marked as missing, the system can use a preset filling method of zero value imputation or keeping the most recent valid value at the corresponding position before participating in the product summation, and at the same time record the missing ratio of that window. When the missing ratio exceeds the upper limit, the result of this round is directly marked as low confidence to avoid destroying the consistency of related calculations when the denominator is changed. In congested traffic conditions, the driver's facial tension sequence is formed after standardization. The skin conductance response can be formed in a translational segment with a candidate offset of 2.4 seconds. The system multiplies the values ​​by time steps to obtain 0.72, 0.30, and 0.02, sums them to 1.04, and divides by 3 to obtain the cross-correlation coefficient at that offset. After comparing the results with other candidate offsets, 2.4 seconds can be confirmed as the optimal offset position. The purpose of this step is to provide executable details of the relevant metrics in dynamic latency estimation, thereby achieving consistency and verifiability of results in different deployment environments.

[0025] In a preferred embodiment of the present invention, based on the physical timestamp and the dynamic delay time difference, the target physiological low-frequency fluctuation data is elastically shifted forward on the time axis. The high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward are semantically aligned based on the event occurrence state using a time series alignment algorithm to generate semantic time dual-axis synchronous mapping features, including: based on the physical timestamp, subtracting the dynamic delay time difference from the overall time axis of the target physiological low-frequency fluctuation data to generate the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward; Feature vectors of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic shift of time axis are extracted in the same time domain dimension and used as common dimension feature vectors; Euclidean distance between common dimension feature vectors is calculated to construct distance matrix between high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic shift of time axis. The dynamic time warping algorithm is used as the time series alignment algorithm to search for the minimum cumulative distance path in the distance matrix. Based on the minimum cumulative distance path, the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted are copied and stretched in time steps to generate semantic time dual-axis synchronous mapping features aligned in the semantic dimension of the target event occurrence.

[0026] This embodiment provides a semantic-temporal dual-axis synchronous mapping generation step; specifically, it only shifts the entire physiological sequence forward by a dynamic delay time difference, which solves the problem of roughly corresponding peaks, but still cannot eliminate the local misalignment caused by different durations and sampling rates within different modalities; For example, a driver's frown may last for 0.6 seconds, while the rise and fall of skin conductance may last for 4 seconds. Simply shifting the overall position cannot completely map the two into the same emotional process. Therefore, this embodiment further performs dynamic time warping after shifting the overall position. Specifically, the system first subtracts the delay from the time axis of the target physiological low-frequency fluctuation data according to the dynamic delay; for example, if the original physiological peak appears at 12:16:10.900 with a delay of 2.4 seconds, the forward-shifted reference time becomes 12:16:08.500; common feature vectors located in the same time domain dimension are extracted from the behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted; the common features here can be understood as event progression descriptions unified to the same dimension, such as local energy, rate of change, peak-to-valley difference, etc. For ease of explanation, assume that the behavior side extracts three time step vectors. , , Three time step vectors were extracted from the physiological side. , , The system calculates the pairwise Euclidean distances and obtains... Distance matrix; For example, the distance from A1 to B1 is 1, the distance from A2 to B2 is 0, and the distance from A3 to B3 is approximately 1.41. The distances for the remaining positions are calculated in the same way. After obtaining the distance matrix, the system uses dynamic time warping to search for a path with the minimum cumulative distance from the top left to the bottom right. If the optimal path is If the second time step on the behavioral side needs to correspond to the second and third time steps on the physiological side, then the system copies or stretches the corresponding time steps to finally generate the aligned semantic biaxial mapping result. The copying and stretching here do not change the original sampling facts, but establish cross-modal mapping relationships in the semantic alignment coordinate system; for example, a short frown on the behavioral side can be mapped to a longer stress-induced rise process on the physiological side; a slow recovery on the physiological side can also be compressed to correspond to the semantic tail after the behavioral side has ended; the resulting synchronous mapping features retain both physical and semantic time indices, which facilitates subsequent windowing and auditing. As an abnormal situation handling mechanism, if the rate of change of curvature of the optimal path searched by dynamic time warping is greater than the preset smoothness threshold, it indicates that the two modes lack consistency in this event, and the system can assign a low confidence label to this mapping. If one dimension of the shared features is missing, the distance matrix can be recalculated after supplementing it with other dimensions; if the length difference between the behavioral and physiological sides is too large, exceeding the preset alignment ratio, for example... The above preset alignment ratio is the maximum tolerance ratio limit of the high-frequency behavioral signal segment and the target physiological low-frequency fluctuation data segment after the time axis is elastically shifted forward in terms of the number of effective sampling time steps. Then the event can be divided into multiple sub-segments and aligned separately to avoid a short behavior forcibly covering the entire long physiological response. In the aforementioned incident of cutting in line, the driver's frown and increased voice intensity were concentrated within 1 second, while the skin conductance lasted for about 4 seconds from peak to recovery. The system first shifted the overall skin conductance forward by 2.4 seconds, and then, through dynamic time warping, matched the short process of frowning onset—intense tension—expression decline with the long process of skin conductance rise—maintaining a high level—gradual decline in semantic dimension, thus forming a synchronous mapping feature that can simultaneously reflect the three stages of the event's initiation, expansion, and recovery. The purpose of this step is to further refine the coarse-grained time-shift compensation into event-process-level semantic alignment, thereby achieving a fusionable representation between modalities with different sampling rates and different durations. To further explain, the forward-shifted physiological data in this embodiment refers to the target physiological low-frequency fluctuation data after the aforementioned time axis elastic forward shift; while the semantic-temporal dual-axis synchronous mapping feature, synchronous mapping feature, and semantic dual-axis mapping result in this embodiment all refer to the same mapping result object formed after overall forward shift and dynamic time warping; Furthermore, the copying and stretching of time steps only occurs in the semantic alignment index layer or mapping relationship layer, which is used to express the correspondence between a certain behavioral time step and one or more physiological time steps. It does not overwrite the original physical timestamps, original sampled values, and original data records required for auditing, thereby ensuring that the semantic alignment results and physical time references can be preserved simultaneously and are consistent.

[0027] In a preferred embodiment of the present invention, when searching for the path with the minimum cumulative distance in the distance matrix using the dynamic time warping algorithm, the calculation logic of the cumulative distance is as follows: the cumulative distance of the current time step node in the distance matrix is ​​equal to the Euclidean distance between the high-frequency behavior signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, and the minimum value among the following three: the adjacent cumulative distance along the time axis of the high-frequency behavior signal, the adjacent cumulative distance along the time axis of the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, and the adjacent cumulative distance in the diagonal direction.

[0028] This embodiment provides a specific search rule for the minimum cumulative distance path. Specifically, if only dynamic time warping is used to find the optimal path, different implementers may make different conventions on the path transition rules, resulting in inconsistent synchronization mapping results. Especially in the vehicle scenario, edge devices need to process quickly online, while the backend system needs to verify offline. If different accumulation rules are used, the event mapping results will be inconsistent. Therefore, this embodiment further defines the recursive logic of the cumulative distance. Specifically, for any current node in the distance matrix, the system takes the node's own Euclidean distance value, and then selects the minimum cumulative distance from the three adjacent nodes on its left, top, and upper left corner and adds them together to obtain the cumulative distance of the current node; this allows the three mapping methods of extension, compression, or synchronous advancement on the time axis to coexist. Use one To illustrate, assume the distance matrix is ​​as follows: first row: 1, 3, 4; second row: 2, 1, 2; third row: 4, 2, 1; the cumulative distance from the top left corner of the starting point is 1. The subsequent nodes in the first row can only extend from the left, so the cumulative distances are 4 and 8 respectively; the subsequent nodes in the first column can only extend from the top, so the cumulative distances are 3 and 7 respectively; when we reach the second row and second column, the current Euclidean distance is 1, the cumulative distance to the left is 3, the cumulative distance to the top is 4, and the cumulative distance on the diagonal is 1. Taking the minimum value of the three, 1, and adding them together, we get 2. When we reach the third row and third column, the current Euclidean distance is 1. The cumulative values ​​of its left side, top side and diagonal are 4, 4 and 2 respectively. Taking the minimum value of 2 and adding them together, we get 3. It can be seen that the path through the diagonal has the lowest cumulative cost, which corresponds to a more reasonable semantic synchronization relationship. This recursive approach can strike a balance between local similarity and overall path optimization, avoiding the deviation of the entire path from the actual event process due to accidental similarity at one or two time steps. For online systems, dynamic programming can also be used to update the path gradually, reducing the overhead of repeated computation. Furthermore, if a neighboring node is invalid, for example, if there is no left or top node at the boundary position, then only the minimum value is taken from the existing neighbors; If a node cannot be calculated in Euclidean distance due to missing features, its distance value can be set as a preset penalty value to make the path avoid the node as much as possible; if the final cumulative distance of the entire path exceeds the allowed limit, it indicates that the alignment quality of this round is poor, and the confidence of this event can be reduced or directly transferred to the exception handling branch. During the semantic alignment of the same interruption event, the local distance between the most tense expression in the second stage of the behavior side and the rapid increase in the response in the second stage of the forward skin conductance side is the smallest. Therefore, the optimal path will preferentially pass through the corresponding node. While the subsequent facial expressions on the behavioral side have weakened, the skin conductance remains high. The pathway allows for an extension along the physiological timeline, thus this sustained high level can be regarded as the tail end of the same emotional event, rather than a new independent event. The purpose of this step is to clarify the path search rules in the synchronization mapping, so as to achieve stable reproduction of semantic alignment results and engineering consistency.

[0029] In a preferred embodiment of the present invention, an elastic sliding window is constructed based on the semantic-temporal dual-axis synchronous mapping feature. The elastic sliding window is used to perform layered slicing of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic forward shift of the time axis, generating an independent semantically aligned multimodal feature matrix, including: extracting the signal energy envelope from the semantic-temporal dual-axis synchronous mapping feature; identifying the energy rise start point and energy decay end point in the signal energy envelope; The starting point of energy rise and the ending point of energy decay are used as the dynamic boundary of the elastic sliding window. The dynamic boundary is used to extract cross-modal data in the semantic-temporal dual-axis synchronous mapping feature to generate an independent semantically aligned multimodal feature matrix corresponding to the starting point of the target event.

[0030] This embodiment provides a step for constructing an elastic sliding window based on the energy envelope. Specifically, after semantic alignment, if a fixed-length window is still used, for example, uniformly truncating 2 seconds before and after the event start point, two types of problems will occur: First, short-term events are divided into excessively large windows, containing too many irrelevant and stable segments; second, long-term physiological recovery processes are truncated by the window, causing a complete emotional process to be broken into multiple segments. Therefore, this embodiment automatically determines the window boundaries based on the energy changes after synchronous mapping. Specifically, the system extracts the signal energy envelope from the semantic-temporal dual-axis synchronous mapping features; the energy here can be obtained by combining the amplitude, rate of change, or local response intensity of the multimodal features at the same semantic time step. Furthermore, the energy envelope after a certain alignment can be simplified to 0.1, 0.2, 0.7, 0.9, 0.6, 0.3, 0.1; the system identifies the position where the energy rises significantly as the starting point of the ascent, such as the position where the energy jumps from 0.2 to 0.7; Simultaneously, the position where the decay falls back and approaches the stable baseline is identified as the decay endpoint, such as the position where the decay falls back from 0.3 to 0.1 and remains stable; thus, the dynamic window boundary of this event is obtained. During window capture, the system does not retain only one modality, but simultaneously captures behavioral data and physiological data after the shift within the range covered by the dynamic boundary, generating an independent semantically aligned multimodal feature matrix; If there are a total of 6 semantic time steps after alignment within the window, and the behavioral modality extracts 2D features for each step, and the physiological modality extracts 2D features for each step, then a [presumably a specific feature set] can be formed. The matrix; if speech sub-features are subsequently added, the matrix width can be expanded to... The matrix here is essentially a data packet of an emotional event, which can be directly input into the backend neural network. The meaning of layered slicing is that the system can simultaneously retain the initiation layer, enhancement layer, and recovery layer of an event; for example, the first two rows of the matrix mainly reflect the sudden changes in facial expressions and speech, the middle two rows reflect the common high position of physiological response and behavioral tension, and the last two rows reflect the physiological recovery after the behavior is relieved; this is more conducive to the classification model to distinguish different states such as instantaneous fright and persistent agitation than simply cutting a fixed segment. As a mechanism for handling abnormal situations, if there are multiple local peaks in the energy envelope and the interval between peaks is less than the preset minimum interval threshold, the system can merge them into the same event window according to the minimum interval threshold; if the interval between peaks exceeds the threshold, it will be split into multiple independent windows and matrices will be built separately. If the start and end points are difficult to identify stably, for example, if the envelope remains high without significant decline, the maximum window length can be used as a forced right boundary to prevent the event from extending indefinitely; if the energy never exceeds the activation threshold, the data segment will not generate an event matrix and will only be retained or discarded as a background segment. Following the incident of cutting in line, the driver's facial and vocal responses rapidly increased, and the skin conductance, after delay compensation, also corresponded to this on the semantic time axis. The system extracted an envelope that first rapidly increased and then slowly decreased, thus positioning the energy rise starting point at the slight pre-tension before the frown appeared and the energy decay endpoint at the point where the skin conductance basically returned to a stable state. The resulting cross-modal matrix completely covered the entire process of the start, peak, and recovery of this agitation event. The purpose of this step is to cut out independent data packets according to the actual duration of the event, thereby achieving a structured representation of the complete emotional process and reducing information redundancy and truncation errors caused by fixed windowing.

[0031] In a preferred embodiment of the present invention, the semantically aligned multimodal feature matrix is ​​input into a preset neural network model to output an emotional state classification result, and the length parameter of the aforementioned preset time window is dynamically updated based on the emotional state classification result to form a data processing closed loop, including: a preset classification confidence threshold; inputting the semantically aligned multimodal feature matrix into a preset neural network model to output an emotional state classification result and the corresponding classification confidence score; If the classification confidence score is not less than the classification confidence threshold, the sentiment state classification result is deemed valid, the sentiment state classification result is output, and the length parameter of the preset time window is updated using the sentiment state classification result. If the classification confidence score is less than the classification confidence threshold, the sentiment state classification result is deemed invalid, triggering an anomaly marking process. The length parameter of the preset time window is increased as the search range for recalculation until the classification confidence score is not less than the classification confidence threshold, or the number of recalculations reaches the preset maximum number of iterations. If the number of recalculations reaches the preset maximum number of iterations and the classification confidence score is still less than the classification confidence threshold, the current semantically aligned multimodal feature matrix is ​​discarded, and an anomaly mark indicating that the sentiment state cannot be identified is output.

[0032] This embodiment provides a closed-loop processing step for updating window parameters in reverse based on classification results. Specifically, in the aforementioned processing, although dynamic delay and elastic sliding window can generate better inputs, if the search window length is fixed for a long time, it may still be unable to adapt to individual differences and state transitions. For example, the same driver's physiological reaction is faster when mentally stressed, and slower when fatigued; if the system does not adjust the window length according to the actual classification effect, the initial parameters may gradually become mismatched; therefore, this embodiment drives parameter updates through classification confidence. Specifically, the system presets a classification confidence threshold, such as 0.80; each time a semantically aligned multimodal feature matrix is ​​generated, it is fed into the neural network model to output the sentiment state classification result and its corresponding score. Furthermore, if a certain matrix is ​​judged to be high stress and agitation, with a score of 0.87, it indicates that the event boundary, delay compensation, and windowing results corresponding to the matrix are relatively reasonable. At this point, the system can record the preset time window length used in this round as a valid sample, and make a small update to the subsequent default window under the same conditions, such as updating it from 4 seconds to 4.5 seconds or keeping it unchanged at 4 seconds; If the score is below the threshold, for example, only 0.62, it indicates that the current input matrix may not fully cover the real emotional process, or the window is too short, causing the physiological peak to not be included, or the window is too long, introducing interference from adjacent events. At this time, the system triggers the anomaly marking process, and after appropriately increasing the search window length, it re-executes the delayed calculation, forward alignment, and window classification. For example, if the score is 0.62 when using a 4-second window for the first time, 0.78 when using a 5-second window for the second time, and 0.84 when using a 6-second window for the third time, then the third result is accepted, and 6 seconds is taken as the reference window length for the driver in the current state family. If the threshold is not reached even after the maximum number of iterations, then the event segment is marked as low confidence and is not used for parameter optimization. Only the original record is retained for subsequent manual review or background model training. A simplified closed-loop deduction can be used: the initial window length is 4 seconds. After classification, a certain event is found to be stressful (0.58); the system expands to 5 seconds and finds agitated (0.73); further expands to 6 seconds and finds agitated (0.86). It can be seen that the original 4-second window does not cover enough physiological tails, making it difficult for the model to form a stable judgment, while the 6-second window is more suitable for this event; the system then corrects the default parameters for subsequent similar events accordingly. As an abnormal situation handling mechanism, if multiple consecutive events are below the threshold, in addition to expanding the window, the system can also check the noise label and sensor status to determine whether it is due to a decline in acquisition quality rather than parameter mismatch. If the confidence continues to decrease after expanding the window, it indicates that too many irrelevant segments have been introduced, and the system can revert to the previous optimal window length. If there are still no effective results after reaching the maximum number of iterations, an abnormality marker is output instead of forcibly giving a highly reliable conclusion to avoid false warnings. During continuous driving at night, drivers experience a slower emotional response due to fatigue and traffic congestion. The system initially uses a 4-second search window commonly used during the day, resulting in uncertain output states from the classification model and a confidence level lower than the preset effective judgment threshold. The system gradually expanded the window to 6 seconds to fully incorporate the slower-appearing skin conductance peaks, which significantly improved the model confidence and stabilized the output of fatigue and irritability. Based on this, the system increased the default window length for the driver under nighttime conditions for continuous monitoring over the next few tens of minutes. The purpose of this step is to allow the preprocessing parameters to be updated adaptively with the classification results, thereby achieving individualized and condition-based continuous optimization, and providing a clear anomaly handling mechanism when the confidence level is insufficient. To further explain, the updated preset time window length parameter in this embodiment belongs to the same parameter object as the search window length used to search for the peak of physiological response around the starting point of the target event. After the update, it is directly fed back to the dynamic delay time difference calculation stage for continued use, rather than generating a separate set of independent window parameters. The default window length, reference window length, and subsequent default window mentioned in the text are all expressions of the same length parameter at different running stages. Correspondingly, when the classification confidence is lower than the threshold and triggers the increase of the preset time window length parameter as the search range for recalculation, the same search window length parameter is also increased. The original process of delayed calculation—flexible forward shift of the time axis—semantic alignment—flexible sliding window—classification evaluation is executed in a loop to ensure that the processing path of the technical solution remains consistent before and after the loop is closed.

[0033] In a preferred embodiment of the present invention, the multimodal heterogeneous data stream is a data stream collected from a medical monitoring device or a driver monitoring system; high-frequency behavioral signals include facial micro-expression video frame data and speech audio stream data; low-frequency physiological signals include electrocardiogram signals, skin conductance signals and electroencephalogram signals.

[0034] This embodiment provides a specific data source configuration method; specifically, in order to make the above synchronous acquisition and preprocessing mechanism feasible in a real system, this embodiment limits the data stream to the data collected by medical monitoring equipment or driver monitoring system, and clarifies the typical composition of high-frequency behavioral signals and low-frequency physiological signals; Specifically, in a driver monitoring system, facial micro-expression video frame data usually comes from infrared cameras located above the steering wheel or in the dashboard area, used to extract high-frequency behavioral features such as changes in the distance between eyebrows, opening and closing of the eye openings, tightness of the corners of the mouth, and head micro-postures; voice audio stream data usually comes from microphones in the roof or A-pillar area, used to extract high-frequency behavioral features such as speech rate, fundamental frequency fluctuations, and short-term energy. Electrocardiogram (ECG) signals, electrodermal (ED) signals, and electroencephalogram (EEG) signals are obtained from steering wheel electrodes, wristbands, electrode patches, or head-mounted devices, respectively, and are used to reflect slower physiological responses at the autonomic and central nervous system levels. In medical monitoring equipment, the deployment methods may differ, but the underlying processing logic remains the same; for example, ward terminals can collect patients' facial videos, voice communication segments, electrocardiograms, electrodermal conductance and electroencephalogram sequences, and use this solution to assist in the identification of anxiety, pain, panic or depression trends. At this time, video and audio still serve as anchors for high-frequency behavioral events, while electrocardiograms, electrodermatography, and electroencephalography still serve as compensations for low-frequency physiological responses and semantic correspondences. Using a simplified data packet example, the video side extracts data at 10 time steps in a given event. Micro-expression features are extracted on the same semantic axis from the speech side. Voice features, electrodermal activity, and electrocardiogram were extracted. Their physiological characteristics can be aligned and pieced together to form... The multimodal feature matrix; If the EEG channel is available, the number of columns can be expanded accordingly. It can be seen that this scheme is scalable in terms of the number of modalities. As long as there is at least one set of high-frequency behavioral signals and at least one set of low-frequency physiological signals, semantic synchronization processing can be completed. As a mechanism for handling abnormal situations, if an EEG sensor is not installed in the actual deployment, the system can continue to operate on the ECG and EKG channels without affecting the overall process; if the voice channel has no effective data for a long time due to the quietness of the vehicle, the system will prioritize facial micro-expressions as behavioral anchors; if the face is unusable due to occlusion but the voice is still effective, the system will switch to voice-driven mode. If both the high-frequency behavioral channel and the low-frequency physiological channel are partially missing, processing will continue only if the minimum modal combination condition is met; otherwise, an insufficient acquisition marker will be output. In the aforementioned hazardous chemical transport vehicles, the driver monitoring system is equipped with an infrared camera, a roof microphone, steering wheel ECG electrodes, and a wristband-type electrodermal module; After the vehicle enters a complex urban road section, the system extracts eyebrow compression and mouth tightening from facial micro-expression video frames, speech acceleration from speech, and delayed physiological elevation from electrocardiogram and skin conductance. After synchronous acquisition, denoising, delay estimation, semantic alignment and elastic sliding window, the above data finally form an event-level multimodal feature matrix that can be used by the classification model. The purpose of this step is to clarify the actual data acquisition terminals and modal components that this solution can be adapted to, so as to realize the engineering implementation from vehicle monitoring to medical monitoring scenarios.

[0035] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for synchronous acquisition and preprocessing of multimodal emotion signals, characterized in that, include: Acquire multimodal heterogeneous data streams, extract high-frequency behavioral signals and low-frequency physiological signals from the multimodal heterogeneous data streams, and obtain the frequency band characteristic parameters of environmental noise accompanying the input of the multimodal heterogeneous data streams, as well as the physical timestamps of the high-frequency behavioral signals and low-frequency physiological signals; Based on the characteristic parameters of environmental noise frequency band, adaptive denoising and baseline calibration are performed on low-frequency physiological signals to separate environmental noise and equipment baseline drift data, while retaining the target physiological low-frequency fluctuation data in the low-frequency physiological signals. The mutation point in the high-frequency behavioral signal is extracted as the starting point of the target event. The mutation point is the data point where the time derivative of the high-frequency behavioral signal exceeds the preset mutation threshold. Based on the preset initial length parameter, a preset time window with the starting point of the target event as the starting boundary is constructed, and the initial length parameter is used as the length parameter of the preset time window. The response peak of the target physiological low-frequency fluctuation data is searched within the preset time window, and the dynamic delay time difference between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data is calculated. Based on the physical timestamp and dynamic delay time difference, the target physiological low-frequency fluctuation data is elastically shifted forward on the time axis. The time series alignment algorithm is used to perform semantic alignment between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, based on the event occurrence state, to generate semantic time dual-axis synchronous mapping features. An elastic sliding window is constructed based on semantic-temporal dual-axis synchronous mapping features. The elastic sliding window is used to perform layered slicing of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic forward shift of the time axis, generating an independent semantically aligned multimodal feature matrix. The semantically aligned multimodal feature matrix is ​​input into a preset neural network model to output the sentiment state classification result. The length parameter of the preset time window is dynamically updated based on the sentiment state classification result, forming a data processing closed loop.

2. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 1, characterized in that, Adaptive denoising and baseline calibration of low-frequency physiological signals are performed based on environmental noise frequency band characteristic parameters. This separates environmental noise and equipment baseline drift data, retaining the target physiological low-frequency fluctuation data in the low-frequency physiological signals, including: Low-frequency physiological signals are decomposed using ensemble empirical mode decomposition algorithm to obtain multi-scale intrinsic mode function components; Based on the frequency characteristics of environmental noise frequency band characteristic parameters and multi-scale intrinsic mode function components, high-frequency intrinsic mode function components of corresponding environmental noise are identified and eliminated. Identify and remove the extremely low-frequency intrinsic mode function components of the corresponding device baseline drift data; The remaining intrinsic mode function components are reconstructed to generate target physiological low-frequency fluctuation data, which includes normal physiological low-frequency fluctuation characteristics caused by biological response inertia.

3. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 2, characterized in that, The abrupt change point in the high-frequency behavioral signal is extracted as the starting point of the target event. Within a preset time window with the starting point of the target event as the initial boundary, the response peak of the target physiological low-frequency fluctuation data is searched. The dynamic delay time difference between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data is calculated, including: Calculate the time derivative of the high-frequency behavioral signal and mark data points whose time derivative exceeds a preset mutation threshold as the starting point of the target event. A preset time window is constructed by taking the starting point of the target event as the initial boundary and combining it with the length parameter of the preset time window. The cross-correlation function is used to calculate the cross-correlation sequence between high-frequency behavioral signals and low-frequency physiological fluctuations of the target within a preset time window; Extract the relative time offset corresponding to the maximum peak value in the cross-correlation coefficient sequence, and use the relative time offset as the dynamic delay time difference.

4. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 3, characterized in that, The logic for calculating the cross-correlation coefficients in a cross-correlation coefficient sequence is as follows: Multiply the high-frequency behavioral signals corresponding to each time step within the preset time window with the target physiological low-frequency fluctuation data after being shifted by the relative time offset, sum the products over all time steps, and then divide by the length of the preset time window to obtain the cross-correlation coefficient. The preset time window is determined by the starting point of the target event and the length of the preset time window.

5. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 4, characterized in that, Based on the physical timestamp and dynamic delay time difference, the target physiological low-frequency fluctuation data is elastically shifted forward on the time axis. A time series alignment algorithm is then used to semantically align the high-frequency behavioral signals with the elastically shifted target physiological low-frequency fluctuation data based on the event occurrence state, generating semantic-temporal dual-axis synchronous mapping features, including: Based on the physical timestamp, the dynamic delay time difference is subtracted from the overall time axis of the target physiological low-frequency fluctuation data to generate the target physiological low-frequency fluctuation data after the time axis is flexibly forwarded. The feature vectors of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic shift of the time axis are extracted in the same time domain dimension and used as the common dimension feature vector. Calculate the Euclidean distance between the feature vectors of the common dimension, and construct the distance matrix between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward; The dynamic time warping algorithm is used as a time series alignment algorithm to search for the path with the minimum cumulative distance in the distance matrix; Based on the minimum cumulative distance path, the high-frequency behavioral signals and the target physiological low-frequency fluctuation data after the time axis is elastically shifted are copied and stretched in time step to generate semantic-temporal dual-axis synchronous mapping features aligned in the semantic dimension of the target event occurrence.

6. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 5, characterized in that, When using the dynamic time warping algorithm to search for the path with the minimum cumulative distance in the distance matrix, the logic for calculating the cumulative distance is as follows: The cumulative distance of the current time step node in the distance matrix is ​​equal to the Euclidean distance between the high-frequency behavioral signal and the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, and the minimum of the following three values: the adjacent cumulative distance along the time axis of the high-frequency behavioral signal, the adjacent cumulative distance along the time axis of the target physiological low-frequency fluctuation data after the time axis is elastically shifted forward, and the adjacent cumulative distance in the diagonal direction.

7. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 6, characterized in that, An elastic sliding window is constructed based on semantic-temporal dual-axis synchronous mapping features. This window is then used to perform layered slicing of high-frequency behavioral signals and target physiological low-frequency fluctuation data after elastic time-axis shift, generating independent semantically aligned multimodal feature matrices, including: Extract the signal energy envelope from the semantic-temporal dual-axis synchronous mapping features; Identify the energy rise start and energy decay end point in the signal energy envelope; The starting point of energy rise and the ending point of energy decay are used as the dynamic boundaries of the elastic sliding window; By utilizing dynamic boundaries, cross-modal data is extracted from semantic-temporal dual-axis synchronous mapping features to generate independent semantically aligned multimodal feature matrices corresponding to the starting point of the target event.

8. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 7, characterized in that, The semantically aligned multimodal feature matrix is ​​input into a pre-defined neural network model to output sentiment state classification results. Based on these results, the length parameter of the pre-defined time window is dynamically updated, forming a closed-loop data processing mechanism including: Preset classification confidence threshold; Input the semantically aligned multimodal feature matrix into a pre-defined neural network model, and output the sentiment state classification result and the corresponding classification confidence score; If the classification confidence score is not less than the classification confidence threshold, the sentiment state classification result is deemed valid, the sentiment state classification result is output, and the length parameter of the preset time window is updated using the sentiment state classification result. If the classification confidence score is less than the classification confidence threshold, the sentiment state classification result is deemed invalid, triggering an anomaly marking process. The length parameter of the preset time window is increased as the search range for recalculation until the classification confidence score is not less than the classification confidence threshold, or the number of recalculations reaches the preset maximum number of iterations. If the number of recalculations reaches the preset maximum number of iterations and the classification confidence score is still less than the classification confidence threshold, the current semantically aligned multimodal feature matrix is ​​discarded, and an anomaly mark indicating that the sentiment state cannot be identified is output.

9. The method for synchronous acquisition and preprocessing of multimodal emotion signals according to claim 8, characterized in that, Multimodal heterogeneous data streams are data streams collected from medical monitoring equipment or driver monitoring systems; High-frequency behavioral signals include facial micro-expression video frame data and speech audio stream data; Low-frequency physiological signals include electrocardiogram (ECG) signals, skin conductance signals, and electroencephalogram (EEG) signals.

Images