Carsickness identification method and device, storage medium and equipment

Abstract

The invention provides a carsickness identification method and device, a storage medium and equipment, and the method comprises the steps: training a teacher model for predicting the carsickness state of a driver and passengers through a multi-source signal sample, training a student model based on a soft label provided by the trained teacher model and an IMU signal sample through a knowledge distillation technology, and carrying out the prediction of the carsickness state of the driver and passengers. And determining the trained student model as a carsickness identification model, deploying the carsickness identification model on a vehicle-mounted chip, and processing the vehicle IMU signal acquired in real time through the carsickness identification model to obtain the carsickness state of the driver and passengers of the vehicle. Therefore, the teacher model based on the multi-source signal is compressed into the student model only needing IMU signal input, so that the problems of limited signal acquisition and insufficient vehicle end computing power in practical application are solved, and real-time and accurate identification of the carsickness state is realized.

Description

Technical Field

[0001] This application relates to the field of automotive technology, and more specifically, to a motion sickness detection method, device, storage medium, and equipment. Background Technology

[0002] Motion sickness is a common physiological discomfort experienced during vehicle travel, especially on long journeys, in complex road conditions, or among certain susceptible individuals. Some high-end vehicles are equipped with air conditioning and seat comfort adjustments, which can alleviate motion sickness symptoms to some extent, but they cannot dynamically sense and respond to the occupants' motion sickness status in real time. Currently, motion sickness recognition typically relies on the fusion processing of multi-source signals, encompassing physiological signals such as electrocardiogram (ECG), electroencephalogram (EEG), and electrodermal conductance, as well as vehicle and environmental road signals. This type of method improves the recognition accuracy of deep learning models by integrating multi-dimensional information, and its performance is closely related to the diversity of signal sources. However, in real-world scenarios, this method suffers from limitations in signal acquisition and insufficient vehicle-side computing power, thus restricting the effective application of motion sickness recognition technology in real-world vehicle environments. Summary of the Invention

[0003] The purpose of this application is to provide a motion sickness recognition method, device, storage medium and equipment, which aims to solve the problems of motion sickness recognition methods in related technologies, which require the model to collect signals from multiple sources at the same time, making it difficult to deploy the model in practical applications, and the model requires a large amount of computing power, making it difficult to achieve real-time recognition in the limited computing power environment of the vehicle.

[0004] In a first aspect, this application provides a motion sickness recognition method, comprising: training a teacher model using multi-source signal samples; the teacher model being used to predict the motion sickness state of the driver and passengers based on the input multi-source signals; training a student model based on soft labels and IMU signal samples provided by the trained teacher model using knowledge distillation technology, and determining the trained student model as the motion sickness recognition model; acquiring the IMU signal of the inertial measurement unit mounted on the vehicle in real time, and inputting the IMU signal into the motion sickness recognition model to obtain the motion sickness state of the driver and passengers of the vehicle.

[0005] In the above implementation process, a teacher model for predicting motion sickness in drivers and passengers is trained using multi-source signal samples. Then, a student model is trained based on the soft labels of the trained teacher model and IMU signal samples using knowledge distillation technology. This trained student model is designated as the motion sickness recognition model. After being deployed on an in-vehicle chip, the motion sickness recognition model processes the real-time acquired vehicle IMU signals to obtain the motion sickness status of the vehicle's drivers and passengers. In this way, the multi-source signal-based teacher model is compressed into a student model that only requires IMU signal input, thus solving the problems of limited signal acquisition and insufficient vehicle-side computing power in practical applications, and achieving real-time and accurate recognition of motion sickness.

[0006] Furthermore, in some examples, the teacher model includes a fusion module, multiple first main feature extraction modules, a first final transpose module, and a first classifier; the fusion module is used to fuse signals from different measurement points in the multi-source signal samples to obtain a multi-source fused representation signal; the multiple first main feature extraction modules are used to extract features step by step from the multi-source fused representation signal; the first final transpose module is used to flatten the fused features extracted by the multiple first main feature extraction modules into a one-dimensional feature vector; the first classifier is used to output a prediction result for the motion sickness state of the driver and passengers based on the one-dimensional feature vector.

[0007] In the above implementation process, a model structure design method for the teacher model is provided, which enables the teacher model to accurately predict the motion sickness state of drivers and passengers based on multimodal input information.

[0008] Furthermore, in some examples, the motion sickness recognition model includes a convolutional sampling module, multiple second principal feature extraction modules, a second final transpose module, and a second classifier; the number of second principal feature extraction modules in the motion sickness recognition model is the same as the number of first principal feature extraction modules in the teacher model; the convolutional sampling module is used to perform dimensionality transformation on the input IMU signal to generate a feature representation of the target dimension; the multiple second principal feature extraction modules are used to extract features step by step from the feature representation; the second final transpose module is used to flatten the IMU features extracted by the multiple second principal feature extraction modules into a one-dimensional feature vector; the second classifier is used to output a prediction result for the motion sickness state of the driver and passengers based on the one-dimensional feature vector.

[0009] In the above implementation process, the motion sickness recognition model, as a synchronous student model, has a network structure that is basically the same as that of the teacher model. The difference is that in the motion sickness recognition model, the IMU signal is first subjected to dimensionality changes through a convolutional sampling module. In this way, the motion sickness recognition model can learn the representation knowledge of motion sickness features in the IMU signal during forward propagation, and at the same time generate a feature sequence with the same structure as the teacher model.

[0010] Furthermore, in some examples, the first main feature extraction module includes multiple encoding modules; the soft label includes the feature encoding results of each encoding module in the teacher model.

[0011] In the above implementation process, the feature encoding results of each encoding module in the teacher model are used as soft labels for synchronous distillation. This can guide the encoding module in the motion sickness recognition model to establish a mapping relationship between IMU signals and physiological signal features, and realize high-quality knowledge transfer from the teacher model to the student model.

[0012] Furthermore, in some examples, during the training process, the motion sickness recognition model calculates a similarity metric function for the features output by each second main feature extraction module and the fused features of the teacher model at the same time step and the same space step, and then backpropagates the calculated metric loss to the preceding second main feature extraction module.

[0013] In the above implementation process, a hierarchical gradient coupling mechanism is designed for PEFM. By establishing a reverse knowledge flow path from deep temporal features to shallow representation units, the motion sickness recognition model learns the time-frequency local features of the temporal signal and gradually transfers the motion sickness feature representation knowledge of the teacher model after encoding the multi-source fusion representation signal to itself. This effectively improves the ability of the motion sickness recognition model to extract features representing early and weak motion sickness symptoms from a single IMU signal.

[0014] Furthermore, in some examples, the similarity metric function consists of feature geometric orientation constraints, feature geometric space constraints, feature probability consistency constraints, feature scale constraints, and maximum mean difference loss; the feature geometric orientation constraints constrain the consistency of feature orientations by calculating the cosine angle between feature vectors; the feature geometric space constraints constrain local similarity by minimizing the geometric distance between fused features and IMU features in the feature space; the feature probability consistency constraints constrain the statistical independence between feature channels by quantifying the KL divergence; and the feature scale constraints constrain the overall energy distribution of feature vectors.

[0015] In the above implementation process, when measuring the similarity between the IMU features of the motion sickness recognition model and the fused features of the teacher model, a complete constraint system for feature knowledge transfer is constructed by fusing feature geometric direction constraints, feature geometric space constraints, feature probability consistency constraints, feature scale constraints, and maximum mean difference loss.

[0016] Furthermore, in some examples, both the first main feature extraction module and the second main feature extraction module include a feature processing unit; the feature processing unit includes a differential progressive embedding module, a feature branching module, a first multi-head attention mechanism module, a feature merging module, a second multi-head attention mechanism module, and a multilayer perception mechanism module; the differential progressive embedding module is used to perform preliminary feature extraction and dimensionality reduction on the input features to obtain a feature map; the feature branching module is used to divide the feature map into a first sub-feature map and a second sub-feature map on an average basis; the first multi-head attention mechanism module is used to perform global feature encoding on the first sub-feature map using a multi-head self-attention mechanism based on a sliding window; the feature merging module is used to concatenate the second sub-feature map and the first sub-feature map after global feature encoding to obtain a merged feature map; the second multi-head attention mechanism module is used to perform secondary processing on the merged feature map using an attention mechanism based on a sliding window to obtain a target feature representation; the multilayer perception mechanism module is used to perform nonlinear transformation on the target feature representation through a combination of residual connections, layer normalization, and feedforward networks.

[0017] In the above implementation process, a fast attention mechanism based on sliding window and branch network is introduced into the PFEM module. Through the self-attention mechanism, the temporal correlation of long sequence signals can be effectively captured. The feature splitting strategy ensures that sufficient temporal information can be obtained in each self-attention calculation by retaining temporal feature information. In this way, not only is the model convergence efficiency and stability effectively improved, but the algorithm complexity is also effectively reduced by reducing redundant calculations.

[0018] Secondly, this application provides a motion sickness recognition device, comprising: a training module for training a teacher model using multi-source signal samples; the teacher model for predicting the motion sickness state of the driver and passengers based on the input multi-source signals; a distillation module for training a student model based on soft labels and IMU signal samples provided by the trained teacher model using knowledge distillation technology, and determining the trained student model as the motion sickness recognition model; and a recognition module for acquiring the IMU signal of the inertial measurement unit mounted on the vehicle in real time, inputting the IMU signal into the motion sickness recognition model to obtain the motion sickness state of the driver and passengers of the vehicle.

[0019] Thirdly, this application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method described in any of the first aspects.

[0020] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0021] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0022] Other features and advantages disclosed in this application will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the above-described technology disclosed in this application.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a motion sickness recognition method provided in this application embodiment; Figure 2 A schematic diagram of the synchronous distillation recognition framework proposed in a non-contact motion sickness detection scheme based on model distillation fusion provided in an embodiment of this application; Figure 3 A block diagram of a motion sickness detection device provided in an embodiment of this application; Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] Motion sickness refers to symptoms such as dizziness, nausea, and vomiting that occur when traveling by vehicle. Some cars are equipped with air conditioning and seat comfort adjustments, which can alleviate motion sickness symptoms to some extent. However, existing systems rely on driver and passenger intervention and cannot dynamically sense and respond to the motion sickness status of passengers in real time. Currently, the method for identifying whether a user is experiencing motion sickness typically uses deep learning models to fuse physiological signals such as the user's electrocardiogram (ECG), electroencephalogram (EEG), and electrodermal conductance (EDC) with vehicle and environmental road signals. The accuracy of the model is highly dependent on the dimensionality of the input information, i.e., the diversity of signal sources. However, in real-world scenarios, especially in in-vehicle systems, continuously collecting multi-source physiological signals presents significant challenges. For example, passengers need to wear multiple sensor devices, affecting passenger comfort, and the limited computing power of the vehicle makes it difficult to support the real-time operation of complex models. Therefore, such methods suffer from limitations in signal acquisition and high computational resource requirements, hindering the effective application of motion sickness recognition technology in real-world vehicle environments.

[0029] To address the aforementioned issues, this application provides a motion sickness recognition scheme. It utilizes multi-source signal samples to train a teacher model for predicting motion sickness in drivers and passengers. Through knowledge distillation, a student model is trained based on soft labels and IMU signal samples provided by the trained teacher model, resulting in a motion sickness recognition model. The vehicle's IMU signals are acquired in real-time and processed by the motion sickness recognition model to determine the motion sickness status of the drivers and passengers. In this way, the multi-source signal-based teacher model is compressed into a student model requiring only IMU signal input, thus solving the problems of limited signal acquisition and insufficient vehicle-side computing power in practical applications, achieving real-time and accurate recognition of motion sickness.

[0030] The embodiments of this application will be described below: like Figure 1 As shown, Figure 1 This is a flowchart of a motion sickness recognition method provided in an embodiment of this application. The method includes: Step 101: Train a teacher model using multi-source signal samples; the teacher model is used to predict the motion sickness status of drivers and passengers based on the input multi-source signals. The multi-source signals mentioned in this step can include physiological signals such as electrocardiogram (ECG), electroencephalogram (EEG), and electrodermal signal, as well as vehicle signals and environmental road signals. The multi-source signal samples can be obtained during the testing phase by collecting physiological signals from drivers and passengers through contact devices, and by collecting vehicle signals and environmental road signals through onboard sensors. In this embodiment, the teacher model is trained based on multi-source signal samples. Since physiological signals are a direct reflection of the subject's physiological state and can characterize the individual differences in motion sickness behavior among different subjects, a teacher model that uses physiological signals as the primary data source and integrates other signals can achieve high recognition accuracy.

[0031] In some embodiments, the multi-source signal samples mentioned in this step have real labels; the real labels are used to indicate the degree of motion sickness of the driver and passengers; accordingly, training the teacher model using multi-source signal samples mentioned in this step may include: inputting the multi-source signal samples into the teacher model to obtain prediction results; calculating cross-entropy loss based on the prediction results and the real labels; and updating the model parameters of the teacher model according to the cross-entropy loss. That is, when collecting multi-source signal samples, the degree of motion sickness of the driver and passengers can be obtained based on their subjective assessment of their physical condition using the MISC (Misery Scale), and then real labels can be added to the samples through manual annotation. During the teacher model training process, the cross-entropy loss is calculated based on the model prediction results of the multi-source signal samples and the corresponding real labels. This cross-entropy loss measures the difference between the model prediction results and the real labels. Based on the calculated cross-entropy loss, the gradient can be calculated using the backpropagation algorithm, and the optimizer parameters can be used. Thus, the finally trained teacher model can accurately predict the motion sickness state of the driver and passengers.

[0032] In some embodiments, the teacher model mentioned in this step may include a fusion module, multiple first main feature extraction modules, a first final transpose module, and a first classifier; the fusion module is used to fuse signals from different measurement points in the multi-source signal samples to obtain a multi-source fusion representation signal; the multiple first main feature extraction modules are used to extract features step by step from the multi-source fusion representation signal; the first final transpose module is used to flatten the fusion features extracted by the multiple first main feature extraction modules into a one-dimensional feature vector; the first classifier is used to output a prediction result for the motion sickness state of the driver and passengers based on the one-dimensional feature vector. In other words, since multi-source signal samples are data collected from multiple sensor devices, they contain signals from multiple measurement points. For these samples, the teacher model uses a fusion module to fuse the signals from different measurement points, such as concatenating the signals from different measurement points along the channel dimension to obtain a multi-source fused representation signal. Then, features are extracted step-by-step through a multi-layer PFEM (Primary Feature Extraction Module) to obtain fused features. These fused features can be considered the most effective high-dimensional features for classification tasks. Finally, a final transpose module flattens the high-dimensional features into a one-dimensional feature vector, which is then output as a prediction vector by the classifier. This allows the teacher model to accurately predict the motion sickness state of drivers and passengers based on multimodal input information.

[0033] Step 102: Using knowledge distillation technology, train the student model based on the soft labels and IMU signal samples provided by the trained teacher model, and determine the trained student model as the motion sickness recognition model. Knowledge distillation, also known as model distillation, is an artificial intelligence model compression technique based on a teacher-student model. It achieves the goal of reducing deployment costs and improving inference efficiency by transferring knowledge from a large teacher model to a small student model. In this embodiment, considering that IMU (Inertial Measurement Unit) signals are used as monitoring parameters of vehicle posture, their acquisition often does not require additional sensor installation, offering advantages such as convenient and stable acquisition. However, vehicle IMU signals can only reflect changes in vehicle posture, while motion sickness is the result of multiple factors. Therefore, relying solely on IMU data to build a motion sickness recognition model makes it difficult to effectively extract features characterizing motion sickness. Based on this, this embodiment proposes a cross-signal synchronous distillation recognition framework. It trains a teacher model using multi-source signals as data sources and a student model using IMU signals as data sources, and achieves a lightweight and high-accuracy design for the motion sickness recognition model through knowledge distillation transfer learning.

[0034] In some embodiments, the motion sickness recognition model mentioned in this step may include a convolutional sampling module, multiple second principal feature extraction modules, a second final transpose module, and a second classifier. The number of second principal feature extraction modules in the motion sickness recognition model is the same as the number of first principal feature extraction modules in the teacher model. The convolutional sampling module is used to perform dimensional transformation on the input IMU signal to generate a feature representation of the target dimension. The multiple second principal feature extraction modules are used to extract features from the feature representation step by step. The second final transpose module is used to flatten the IMU features extracted by the multiple second principal feature extraction modules into a one-dimensional feature vector. The second classifier is used to output a prediction result for the motion sickness state of the driver and passengers based on the one-dimensional feature vector. That is, the motion sickness recognition model, as a synchronous student model, has a network structure that is basically the same as that of the teacher model. The difference is that in the motion sickness recognition model, the IMU signal is first transformed by the convolutional sampling module to generate a feature representation of the target dimension, where the target dimension is the dimension corresponding to the multi-source fusion representation signal. In this way, the motion sickness recognition model can learn the representation knowledge of motion sickness features in the IMU signal during forward propagation, and at the same time generate a feature sequence with the same structure as the teacher model.

[0035] In some embodiments, the aforementioned first main feature extraction module includes multiple encoding modules; the soft labels mentioned in this step may include the feature encoding results of each encoding module in the teacher model. That is, unlike the traditional distillation learning method that uses the probability distribution output by the final decoding layer of the teacher model as the soft label, this embodiment uses the feature encoding results of each encoding module in the teacher model as the soft label for synchronous distillation. This guides the encoding modules in the motion sickness recognition model to establish a mapping relationship between IMU signals and physiological signal features, achieving high-quality knowledge transfer from the teacher model to the student model.

[0036] In addition, in traditional distillation learning, the student model only backpropagates through the output layer loss, and the gradient needs to be passed through all layers of the student model, which can easily lead to gradient vanishing or diffusion. Moreover, the student model only learns the decision results of the teacher model and cannot reproduce its reasoning process. Based on this, in some embodiments, the motion sickness recognition model mentioned in this step can calculate the similarity measurement function between the features output by each second main feature extraction module and the fusion features of the teacher model at the same time step and the same space step during the training process, and backpropagate the calculated measurement loss to the front second main feature extraction module. In other words, after each feature extraction by the PFEM, the high-dimensional features of the motion sickness recognition model are compared with the fusion features of the teacher model at the same time step and in the same spatial step using a similarity metric function. This similarity is then backpropagated to the preceding deep learning layers. Taking a motion sickness recognition model with three PFEM layers as an example, the first PFEM layer receives the gradient of the metric loss of the subsequent two PFEM layers with respect to the output features, and then uses this gradient combined with its own calculated metric loss to update the parameters. Similarly, the second PFEM layer receives the gradient of the metric loss of the last PFEM layer with respect to the output features, and then uses this gradient combined with its own calculated metric loss to update the parameters. The last PFEM layer only relies on its own calculated metric loss to update the parameters. In this way, while learning the time-frequency local features of the time-series signal, the motion sickness recognition model gradually transfers the knowledge of motion sickness feature representation after encoding the multi-source fused representation signal from the teacher model to itself, thereby effectively improving the ability of the motion sickness recognition model to extract features representing early and weak motion sickness symptoms from a single IMU signal.

[0037] Furthermore, in some embodiments, the aforementioned similarity metric function can be composed of feature geometric direction constraints, feature geometric space constraints, feature probability consistency constraints, feature scale constraints, and maximum mean difference loss; the feature geometric direction constraints constrain the consistency of feature directions by calculating the cosine angle between feature vectors; the feature geometric space constraints constrain local similarity by minimizing the geometric distance between fused features and IMU features in the feature space; the feature probability consistency constraints constrain the statistical independence between feature channels by quantifying the KL divergence; and the feature scale constraints constrain the overall energy distribution of feature vectors.

[0038] In other words, when measuring the similarity between the IMU features of the motion sickness recognition model and the fused features of the teacher model, a similarity measurement function can be constructed by integrating feature geometric orientation constraints, feature geometric space constraints, feature probability consistency constraints, feature scale constraints, and maximum mean difference loss. Specifically, feature geometric orientation constraints require that the fused features and IMU features maintain similar directional distributions in the feature space, avoiding feature alignment shifts due to network parameter fluctuations. Feature geometric space constraints require that the fused features and IMU maintain proximity and feature morphological similarity in the low-dimensional subspace, preventing nonlinear torque in the feature space during the learning process of excessively deep network layers in deep learning models. This is suitable for capturing the synergistic relationship between high-frequency transient features in IMU signals and low-frequency trend features of multi-source signals. Feature probability consistency constraints and feature scale constraints ensure the amplitude scale stability of the fused features and IMU features under different operating conditions, avoiding feature distortion caused by changes in the operating state of the gyroscope device. Maximum mean difference loss improves classification robustness by maximizing the inter-class mean difference and significantly enhances the fault detection capability of IMU signals under complex operating conditions by mining the nonlinear discrimination boundary in the feature space. In implementation, the similarity measurement function can achieve a balance of multi-objective optimization by weighted fusion of these constraints, thereby constructing a complete constraint system for feature knowledge transfer.

[0039] Furthermore, considering the high computational cost of dual-model simultaneous distillation during the training phase and long-term sequence signal processing, in some embodiments, both the first and second main feature extraction modules include feature processing units. The feature processing unit includes a differential progressive embedding module, a feature branching module, a first multi-head attention mechanism module, a feature merging module, a second multi-head attention mechanism module, and a multilayer perceptron mechanism module. The differential progressive embedding module performs preliminary feature extraction and dimensionality reduction on the input features to obtain a feature map. The feature branching module divides the feature map into a first sub-feature map and a second sub-feature map on average. The first multi-head attention mechanism module uses a sliding window-based multi-head self-attention mechanism to perform global feature encoding on the first sub-feature map. The feature merging module concatenates the second sub-feature map and the globally encoded first sub-feature map to obtain a merged feature map. The second multi-head attention mechanism module uses a sliding window-based attention mechanism to perform secondary processing on the merged feature map to obtain a target feature representation. The multilayer perceptron mechanism module performs a nonlinear transformation on the target feature representation through a combination of residual connections, layer normalization, and feedforward networks.

[0040] In other words, in the knowledge distillation network of this application, each PFEM includes a differential progressive embedding module, a feature branching module, a first multi-head attention mechanism module, a feature merging module, a second multi-head attention mechanism module, and a multilayer perception mechanism module. These modules cooperate sequentially to form a complete feature processing unit. Multiple PFEMs are stacked to form a deep feature extraction network. In implementation, the differential progressive embedding module can be considered as a convolutional block, which performs preliminary feature extraction and dimensionality reduction to prepare a more refined feature map for subsequent more complex processing. The feature branching module can use operations such as torch.chunk() or torch.split() to divide the input feature map into two sub-feature maps on an average scale along the channel dimension, namely the first sub-feature map and the second sub-feature map. The first multi-head attention mechanism module only processes the first sub-feature map. It uses a multi-head self-attention mechanism based on a sliding window, that is, the first sub-feature map is divided into multiple non-overlapping channels, and the attention calculation is only performed within each window, capturing the local up and down within the window. The first feature map is processed by a multi-head attention mechanism, which reduces the computational complexity from quadratic to linear. The second feature merging module concatenates the processed first sub-feature map and the unprocessed second sub-feature map along the channel dimension, fusing local details and global contextual information. The third multi-head attention mechanism module further processes the merged feature map, enabling the model to perform more refined feature learning and relational modeling based on the fused information, further integrating global and local information. The fourth multi-layer perception mechanism module is a deep learning module that integrates residual connections, layer normalization, and feedforward networks. It can perform non-linear transformations on the integrated features and is a self-stabilizing and easily trained feature learning unit. Thus, the self-attention mechanism effectively captures the temporal correlation of long-sequence signals, while the feature splitting strategy preserves temporal contextual information, ensuring that each self-attention computation acquires sufficient temporal information. This design not only effectively improves the model's convergence efficiency and stability but also reduces algorithm complexity by minimizing redundant computations.

[0041] Step 103: Acquire the IMU signal from the inertial measurement unit mounted on the vehicle in real time, input the IMU signal into the motion sickness recognition model, and obtain the motion sickness status of the vehicle's occupants.

[0042] In this embodiment, after training the motion sickness recognition model, the model is deployed on the vehicle chip. In this way, during vehicle operation, the motion sickness recognition model can be used to process real-time IMU signals to achieve accurate identification of the vehicle's occupants.

[0043] In this embodiment, a teacher model for predicting motion sickness in drivers and passengers is trained using multi-source signal samples. A student model is then trained based on soft labels and IMU signal samples provided by the trained teacher model using knowledge distillation technology. This trained student model is designated as the motion sickness recognition model. After being deployed on an in-vehicle chip, the motion sickness recognition model processes the real-time acquired vehicle IMU signals to determine the motion sickness status of the drivers and passengers. This approach compresses the multi-source signal-based teacher model into a student model that only requires IMU signal input, thus solving the problems of limited signal acquisition and insufficient vehicle-side computing power in practical applications, and achieving real-time and accurate recognition of motion sickness.

[0044] To provide a more detailed explanation of the solution in this application, a specific embodiment is described below: This embodiment provides a non-contact motion sickness detection scheme based on model distillation fusion. The scheme proposes, as follows: Figure 2 The synchronous distillation identification framework shown in this paper uses fusion model 21 as the synchronous teacher model and IMU model 22 as the synchronous student model. Figure 2 Solid lines in the diagram represent forward propagation, and dashed lines represent backward propagation.

[0045] Specifically, during the training of the fusion model 21, the multi-source signal samples 23 (with the physiological signals of the subjects as the main data source) are processed by the fusion module 24 to generate a multi-source fused fault representation signal with dimensions B×256×16, where B is the number of samples processed at one time. Features are extracted step by step from the multi-source fused fault representation signal through a three-layer PFEM 25, and then flattened into a one-dimensional feature vector with dimensions B×C×1 by the final transpose module 26, where C is the number of channels. Finally, the classifier outputs a prediction vector, and the cross-entropy loss is calculated based on the prediction vector and the true labels 27 corresponding to the multi-source signal samples. The real label 27 represents 11 levels of motion sickness according to the MISC scale, ranging from 0 to 10, with higher numbers indicating more severe motion sickness. The formula for calculating the cross-entropy loss is... As shown below:

[0046] In the formula, For the first The sample belongs to the first One-hot encoding of a class, The model predicts the first Class probability, This is the general category.

[0047] IMU model 22 also has 3 layers of PFEM (numbered 25 in the figure) and a final transpose module 26. In IMU model 22, the IMU signal 28 is first transformed in dimension by the convolutional sampling module 29 to adapt to the same network structure as the fusion model. During forward propagation, it learns the representation knowledge of motion sickness features in the IMU signal and generates feature sequences with the same structure as the teacher model. After feature extraction in each PFEM layer, the high-dimensional features of IMU model 22 are compared with the fusion features of the fusion model at the same time step and the same spatial step to calculate a high-level similarity metric function. This loss function is then backpropagated to the preceding deep learning layers. The construction process of this high-level similarity metric function includes: First, the high-dimensional fusion features and IMU features are flattened into one-dimensional feature vectors, which can be represented as:

[0048]

[0049] In the formula, This represents the feature vector after encoding and flattening the i-th PFEM in the fusion model 21. This represents the feature vector of the fusion model 21 before the i-th PFEM encoding and flattening. This represents the feature vector after the i-th PFEM encoding and flattening in IMU model 22. This represents the feature vector of the i-th PFEM encoding and flattening in IMU model 22. It is a function that changes dimensions. Indicates the number of samples processed at one time. Indicates the number of channels. Indicates the length of the input sample; Secondly, we construct a high-level similarity metric function, which can be represented as:

[0050] In the formula, This represents a high-level similarity metric function. Indicates characteristic geometric direction constraints. Represents the characteristic geometric space constraints, This indicates a feature probability consistency constraint. Indicates feature scale constraints. This represents the loss due to the maximum mean difference, while , , , , These are the weights corresponding to these constraints. Optionally, each of these weights can be set to 0.2 to achieve a balance in multi-objective optimization; where: Feature geometric orientation constraints Consistency constraints on feature directions are applied by calculating the cosine angle between feature vectors, which can be expressed as:

[0051] In the formula, This represents a matrix operation function used to calculate the determinant of a matrix. Denotes the F-norm of a matrix; Feature geometric space constraints Local similarity constraints are applied by minimizing the geometric distance between the fused features and the IMU features in the feature space, which can be expressed as:

[0052] Feature probability consistency constraint Based on the maximum entropy principle of probability distribution, the statistical independence constraint between feature channels is quantified through KL divergence, which can be expressed as:

[0053] In the formula, This represents the normalized exponential function; Represents matrix operation functions The logarithmic function; Feature Scale Constraint Suppressing sensor noise interference by constraining the overall energy distribution of the eigenvectors can be expressed as:

[0054] Maximum mean difference loss Improving classification robustness by maximizing the difference in inter-class means can be expressed as:

[0055] In the formula, This represents the kernel matrix between samples within the source domain (fusion features). The kernel matrix represents the overlap between samples from the source and target domains (IMU features), indicating the degree of overlap between the two distributions. This represents the kernel matrix between samples within the target domain.

[0056] Advanced similarity measurement functions It achieves multi-time step and multi-dimensional constraints on features. Specifically, it constructs a complete constraint system for feature knowledge transfer layer by layer, from directional consistency (cosine loss), geometric consistency (Euclidean distance) to statistical independence (KL divergence). Norm constraints and maximum mean difference loss are designed for amplitude stability and inter-class separability, respectively, to adapt to special application scenarios with high noise and strong interference of sensor signals under complex and variable working conditions.

[0057] Furthermore, to reduce computational complexity in this simultaneous distillation recognition framework, a fast attention mechanism based on sliding windows and branch networks is designed for feature extraction. This mechanism introduces a branch network into PFEM, dividing the feature map into two sub-maps for parallel processing. One sub-map performs global feature encoding based on a multi-head self-attention mechanism using a sliding window, while the other sub-map skips the first window attention mechanism through a feature-level merging operation, incorporating global contextual feature information. Subsequently, the two sub-maps are concatenated to form a feature map, which is then refined through a secondary window attention mechanism. This branching strategy not only reduces the computational complexity of self-attention from quadratic to linear, but also enables dynamic parameter updates through a cross-branch gradient coupling mechanism.

[0058] Self-attention mechanisms effectively capture the temporal correlations of long-sequence signals, while feature splitting strategies preserve temporal context information, ensuring sufficient temporal information is acquired in each self-attention computation. This design not only significantly improves model convergence efficiency and stability but also reduces algorithm complexity by minimizing redundant computations. Specifically, the computational complexity of the traditional MSA (Multi-head Self-Attention) module is... The computational complexity of the W-MSA (Window Multi-head Self-Attention) module is... The computational complexity of the fast attention mechanism based on sliding window and branch network used in this embodiment is... In the formula This is the input sample length. Indicates the number of channels. This represents the length of the reconstructed sample, i.e., the window size. Since the reconstructed sample length is the same as the sample length, the computational complexity of the fast attention mechanism based on sliding windows and branch networks used in this embodiment is much lower than that of the global attention mechanism.

[0059] This embodiment constructs a fusion feature distillation model (teacher model) with physiological signals as the main data source and a motion sickness feature learning model (student model) with IMU signals as the data source. It designs a cross-signal high-level similarity measurement function and knowledge transfer strategy, and introduces a lightweight attention mechanism to reduce computational complexity. Actual testing confirms that the motion sickness feature learning model established based on this embodiment can achieve highly accurate and timely motion sickness identification of vehicles by relying solely on IMU signals as the data source.

[0060] Corresponding to the embodiments of the aforementioned methods, this application also provides embodiments of a motion sickness detection device and a terminal thereof: like Figure 3 As shown, Figure 3 This is a block diagram of a motion sickness detection device provided in an embodiment of this application. The device includes: Training module 31 is used to train a teacher model using multi-source signal samples; the teacher model is used to predict the motion sickness state of drivers and passengers based on the input multi-source signals. Distillation module 32 is used to train a student model based on soft labels and IMU signal samples provided by a trained teacher model using knowledge distillation technology, and to identify the trained student model as a motion sickness recognition model. The identification module 33 is used to acquire the IMU signal of the inertial measurement unit mounted on the vehicle in real time, input the IMU signal into the motion sickness identification model, and obtain the motion sickness state of the vehicle's driver and passengers.

[0061] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0062] This application also provides an electronic device, please refer to [link to application]. Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device may include a processor 410, a communication interface 420, a memory 430, and at least one communication bus 440. The communication bus 440 is used to enable direct communication between these components. In this embodiment, the communication interface 420 of the electronic device is used for signaling or data communication with other node devices. The processor 410 may be an integrated circuit chip with signal processing capabilities.

[0063] The processor 410 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or the processor 410 can be any conventional processor.

[0064] The memory 430 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 430 stores computer-readable instructions. When these computer-readable instructions are executed by the processor 410, the electronic device can perform the aforementioned operations. Figure 2 The various steps involved in the method implementation examples.

[0065] Alternatively, the electronic device may also include a storage controller and an input / output unit.

[0066] The memory 430, storage controller, processor 410, peripheral interface, and input / output unit are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 440. The processor 410 is used to execute executable modules stored in the memory 430, such as software function modules or computer programs included in electronic devices.

[0067] The input / output unit is used to provide users with the ability to create tasks and to set optional start periods or preset execution times for those tasks, thereby enabling user-server interaction. The input / output unit may be, but is not limited to, a mouse and keyboard.

[0068] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0069] This application also provides a storage medium storing instructions. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described again here.

[0070] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method described in the method embodiment.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0072] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0073] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0075] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0076] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A motion sickness detection method, characterized in that, include: Training a teacher model using multi-source signal samples; The teacher model is used to predict the motion sickness status of drivers and passengers based on the input multi-source signals; Using knowledge distillation technology, a student model is trained based on soft labels and IMU signal samples provided by a trained teacher model, and the trained student model is identified as a motion sickness recognition model. The IMU signal from the inertial measurement unit (IMU) on the vehicle is acquired in real time, and the IMU signal is input into the motion sickness recognition model to obtain the motion sickness status of the vehicle's occupants.

2. The method according to claim 1, characterized in that, The teacher model includes a fusion module, multiple first main feature extraction modules, a first final transpose module, and a first classifier; the fusion module is used to fuse signals from different measurement points in the multi-source signal samples to obtain a multi-source fusion representation signal; the multiple first main feature extraction modules are used to extract features from the multi-source fusion representation signal step by step; The first final transpose module is used to flatten the fused features extracted by the multiple first main feature extraction modules into a one-dimensional feature vector; the first classifier is used to output a prediction result for the motion sickness state of the driver and passengers based on the one-dimensional feature vector.

3. The method according to claim 2, characterized in that, The motion sickness recognition model includes a convolutional sampling module, multiple second main feature extraction modules, a second final transpose module, and a second classifier. The number of second main feature extraction modules in the motion sickness recognition model is the same as the number of first main feature extraction modules in the teacher model. The convolutional sampling module is used to perform dimensionality transformation on the input IMU signal to generate a feature representation of the target dimension. The multiple second main feature extraction modules are used to extract features from the feature representation step by step. The second final transpose module is used to flatten the IMU features extracted by the multiple second main feature extraction modules into a one-dimensional feature vector; the second classifier is used to output a prediction result for the motion sickness state of the driver and passengers based on the one-dimensional feature vector.

4. The method according to claim 3, characterized in that, One of the multi-source signals is a physiological signal; both the first main feature extraction module and the second main feature extraction module include multiple encoding modules; the soft label includes the feature encoding results of each encoding module in the teacher model; the soft label is used to guide the encoding module of the student model to establish a mapping relationship between the features of the IMU signal and the features of the physiological signal.

5. The method according to claim 3, characterized in that, During the training process, the motion sickness recognition model calculates a similarity metric function by combining the features output by each second main feature extraction module with the fusion features of the teacher model at the same time step and the same space step, and then backpropagates the calculated metric loss to the preceding second main feature extraction module.

6. The method according to claim 5, characterized in that, The similarity metric function includes feature geometric direction constraints, feature geometric space constraints, feature probability consistency constraints, feature scale constraints, and maximum mean difference loss. The feature geometric direction constraints constrain the consistency of feature directions by calculating the cosine angle between feature vectors. The feature geometric space constraints constrain local similarity by minimizing the geometric distance between fused features and IMU features in the feature space. The feature probability consistency constraints constrain the statistical independence between feature channels by quantifying the KL divergence. The feature scale constraints constrain the overall energy distribution of feature vectors.

7. The method according to claim 3, characterized in that, Both the first main feature extraction module and the second main feature extraction module include a feature processing unit; The feature processing unit includes a micro-element progressive embedding module, a feature branching module, a first multi-head attention mechanism module, a feature merging module, a second multi-head attention mechanism module, and a multi-layer perception mechanism module; The incremental embedding module is used to perform preliminary feature extraction and dimensionality reduction on the input features to obtain a feature map; the feature branching module is used to divide the feature map into a first sub-feature map and a second sub-feature map on an average basis; the first multi-head attention mechanism module is used to perform global feature encoding on the first sub-feature map using a multi-head self-attention mechanism based on a sliding window. The feature merging module is used to concatenate the second sub-feature map and the first sub-feature map after global feature encoding to obtain a merged feature map; the second multi-head attention mechanism module is used to perform secondary processing on the merged feature map using a sliding window-based attention mechanism to obtain the target feature representation; the multilayer perception mechanism module is used to perform nonlinear transformation on the target feature representation through a combination of residual connections, layer normalization and feedforward networks.

8. A motion sickness detection device, characterized in that, include: The training module is used to train a teacher model using multi-source signal samples; the teacher model is used to predict the motion sickness status of drivers and passengers based on the input multi-source signals. The distillation module is used to train the student model based on the soft labels and IMU signal samples provided by the trained teacher model using knowledge distillation technology, and to identify the trained student model as the motion sickness recognition model. The identification module is used to acquire the IMU signal of the inertial measurement unit mounted on the vehicle in real time, input the IMU signal into the motion sickness identification model, and obtain the motion sickness status of the vehicle's driver and passengers.

9. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 7.

Images