Carsickness identification method, electronic device and vehicle

Abstract

The invention relates to a carsickness recognition method, an electronic device and a vehicle, and the method comprises the steps: obtaining the environment features of the vehicle and the visual features of a target passenger in the vehicle; the vehicle environment characteristics comprise vehicle state parameters and environment parameters associated with inducing carsickness of passengers in the vehicle, and the visual characteristics are behavior performance characteristics related to the carsickness state; constructing a main feature vector according to the visual features; according to the visual features and the vehicle environment features, auxiliary feature vectors are constructed; based on the correlation degree between the main feature vector and the auxiliary feature vectors, performing dynamic weighted fusion on each auxiliary feature vector to generate a multi-modal feature vector; and carsickness identification processing is carried out on the multi-modal feature vector, and a carsickness identification result for the target passenger is generated. According to the invention, the problem of low carsickness identification accuracy is solved.

Description

Technical Field

[0001] This application relates to the field of vehicles, and in particular to motion sickness detection methods, electronic devices, and vehicles. Background Technology

[0002] In car travel scenarios, motion sickness has long been characterized by a significant lag in detection. This is especially true for newborns and young children who lack the ability to express themselves verbally. Their physiological discomfort during motion sickness (such as dizziness, nausea, and vomiting) can only be conveyed through subtle nonverbal cues. Parents often struggle to monitor their children's condition while driving, and they only notice the problem when obvious symptoms such as vomiting or intense crying appear. By then, the child has already endured prolonged physical discomfort, which may trigger a chain reaction of subsequent fear of traveling and gastrointestinal disorders.

[0003] In related technologies, motion sickness detection solutions for children mostly rely on wearable devices (such as smart bracelets and physiological monitoring patches). However, these solutions incur additional costs, requiring the separate purchase of detection devices adapted to children's body shapes, and the devices need regular charging and maintenance. Secondly, young children have low acceptance of foreign objects close to their bodies, and are prone to tearing or resisting wearing them, leading to interruption or failure of detection data, making it difficult to achieve continuous and stable monitoring, thus resulting in low accuracy in motion sickness detection.

[0004] Currently, no effective solution has been proposed to address the issue of low accuracy in motion sickness detection in related technologies. Summary of the Invention

[0005] This application provides a motion sickness recognition method, electronic device, and vehicle to at least address the problem of low accuracy in motion sickness recognition in related technologies.

[0006] In a first aspect, embodiments of this application provide a motion sickness recognition method, the method comprising:

[0007] Acquire vehicle environmental features and visual features of the target occupants inside the vehicle; the vehicle environmental features include vehicle state parameters and environmental parameters associated with inducing motion sickness in passengers inside the vehicle, and the visual features are behavioral characteristics related to motion sickness.

[0008] A main feature vector is constructed based on the visual features; and an auxiliary feature vector is constructed based on the visual features and the vehicle environment features.

[0009] Based on the correlation between the main feature vector and the auxiliary feature vector, the auxiliary feature vectors are dynamically weighted and fused to generate a multimodal feature vector;

[0010] The multimodal feature vector is processed for motion sickness recognition to generate motion sickness recognition results for the target passenger.

[0011] In some embodiments, the motion sickness recognition processing of the multimodal feature vector to generate motion sickness recognition results for the target occupant includes:

[0012] The multimodal feature vectors from the continuous time series are sequentially input into the trained time series feature extraction model;

[0013] The time-series feature extraction model is used to capture the dynamic change pattern of the multimodal feature vector in the continuous time series, and the motion sickness recognition result is generated based on the dynamic change pattern.

[0014] In some embodiments, the multimodal feature vector over the continuous time series includes: the multimodal feature vector of the current frame and the multimodal feature vector of historical frames;

[0015] The step of sequentially inputting multimodal feature vectors from a continuous time series into a trained temporal feature extraction model for temporal processing to generate the motion sickness recognition result includes:

[0016] The multimodal feature vectors on the continuous time series are sequentially input into the temporal feature extraction model. Through the first attention layer in the temporal feature extraction model, the multimodal feature vector of the current frame is used as the first query vector, and the multimodal feature vector of the historical frames is used as the first key value vector. Based on the first query vector, cross-attention calculation is performed on the first key value vector to output the motion sickness recognition result.

[0017] In some embodiments, the first attention layer is a deformable attention layer; the step of performing cross-attention calculation on the first key-value vector based on the first query vector and outputting the motion sickness recognition result includes:

[0018] The sampling offset of the first query vector is dynamically learned through the deformable attention layer, and the key positions in the first key-value vector are extracted based on the sampling offset.

[0019] The motion sickness recognition result is output by weighting and fusing the key locations based on the first query vector.

[0020] In some embodiments, the step of constructing a main feature vector based on the visual features and constructing an auxiliary feature vector based on the visual features and the vehicle environment features includes:

[0021] The visual features and the vehicle environment features are input into the trained multimodal fusion model; the multimodal fusion model includes at least a main branch and an auxiliary branch;

[0022] The visual features are processed by feature projection through the main branch of the multimodal fusion model to output the main feature vector, and the visual features and the vehicle environment features are processed by feature projection through the auxiliary branch of the multimodal fusion model to output the auxiliary feature vector.

[0023] In some embodiments, the multimodal fusion model further includes a second attention layer; the step of fusing the auxiliary feature vectors based on the main feature vector to generate a multimodal feature vector includes:

[0024] The main feature vector is used as the second query vector and the auxiliary feature vector is used as the second key vector via the second attention layer. Attention is calculated on the second key vector based on the second query vector to output the multimodal feature vector.

[0025] In some embodiments, constructing an auxiliary feature vector based on the visual features and the vehicle environment features includes:

[0026] Acquire the voice characteristics of the target occupant;

[0027] The auxiliary feature vector is constructed based on the visual features, the vehicle environment features, and the sound features.

[0028] In some embodiments, obtaining the visual characteristics of the target occupant inside the vehicle includes:

[0029] Acquire image data containing the target occupants inside the vehicle;

[0030] The image data is input into a trained visual feature extraction network, which is used to extract fine-grained features related to motion sickness from the image data.

[0031] The image data is processed sequentially through at least one processing stage component in the visual feature extraction network, which consists of alternating basic convolutional modules and feature enhancement modules, to obtain multi-level fine-grained features. The basic convolutional module is used to extract low-level visual features from the image data, and the feature enhancement module is used to extract multi-level fine-grained features from the low-level visual features by fusing variable convolutional kernels and channel separation strategies.

[0032] The multi-level fine-grained features output by the last-stage processing component in the visual feature extraction network are processed by global pooling and fully connected layers to generate the visual features.

[0033] Secondly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the motion sickness recognition method as described in the first aspect above.

[0034] Thirdly, embodiments of this application provide a vehicle, including a vehicle body and an electronic device as described in the second aspect above.

[0035] Compared to related technologies, the motion sickness recognition method, electronic device, and vehicle provided in this application acquire vehicle environmental features and visual features of the target occupant inside the vehicle. The vehicle environmental features include vehicle state parameters and environmental parameters associated with inducing motion sickness in passengers, and the visual features are behavioral characteristics related to motion sickness. A main feature vector is constructed based on the visual features; and auxiliary feature vectors are constructed based on the visual features and vehicle environmental features. Based on the correlation between the main feature vector and the auxiliary feature vectors, each auxiliary feature vector is dynamically weighted and fused to generate a multimodal feature vector. The multimodal feature vector is then processed for motion sickness recognition to generate a motion sickness recognition result for the target occupant.

[0036] By employing the above methods, the system can proactively filter out the key evidence most relevant to the current motion sickness state from multimodal information (e.g., dynamically linking a "frowning" expression with the acceleration of "severe jolts"), while automatically suppressing irrelevant or interfering information (such as similar actions caused by normal play). This generates a highly discriminative fusion feature, enabling the system to perform complete detection in the early stages before motion sickness symptoms fully manifest (e.g., vomiting, violent crying), relying solely on existing in-vehicle equipment without the need for any additional hardware. This allows for timely and accurate capture of subtle, progressive signs of discomfort, achieving high-precision, early detection, and strong anti-interference intelligent identification of motion sickness in vehicle occupants (especially young children). This fundamentally avoids the "delay in motion sickness detection" caused by children's insufficient language ability and parents' distracted driving, thus effectively solving the problem of low accuracy in motion sickness identification.

[0037] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1This is a hardware structure block diagram of a terminal for a motion sickness recognition method according to an embodiment of this application;

[0040] Figure 2 This is a flowchart of a motion sickness recognition method according to an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the structure of a visual feature extraction network according to an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the structure of a multimodal fusion model according to an embodiment of this application;

[0043] Figure 5 This is a flowchart of another motion sickness recognition method according to an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0045] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0046] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of a terminal for a motion sickness recognition method according to an embodiment of this application. Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0048] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the motion sickness recognition method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0049] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0050] This embodiment provides a motion sickness detection method. Figure 2 This is a flowchart of a motion sickness recognition method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0051] Step S210: Obtain vehicle environmental features and visual features of the target occupants inside the vehicle; vehicle environmental features include vehicle state parameters and environmental parameters associated with inducing motion sickness in passengers inside the vehicle, and visual features are behavioral characteristics related to motion sickness.

[0052] The aforementioned vehicle environmental characteristics refer to physical parameters strongly correlated with motion sickness according to medical and physiological studies. These parameters can include: vehicle state parameters, primarily referring to the vehicle's kinematic data in three-dimensional space, including but not limited to longitudinal, lateral, and vertical accelerations (commonly known as sudden braking, sharp turns, and bumps), as well as the frequency and amplitude of vehicle vibrations. These parameters directly affect the passenger's vestibular and visual systems, and are the root cause of motion perception conflict. For example, continuous low-frequency lateral swaying can easily induce nausea. In addition, the aforementioned vehicle environmental characteristics can also include another category: in-vehicle environmental parameters, which encompass the microenvironmental state of the enclosed space of the vehicle, mainly including temperature and humidity. High temperature and high humidity environments exacerbate human discomfort, creating a "cumulative effect" with motion stimulation and accelerating the onset of motion sickness symptoms.

[0053] It is important to note that the acquisition of these vehicle environmental characteristic parameters relies entirely on the vehicle's existing sensor network, such as the inertial measurement unit (IMU), body CAN bus data, and temperature and humidity sensors; therefore, no additional hardware is required. For example, longitudinal acceleration... lateral acceleration Vertical acceleration and frequency of bumps These are all "dynamic state parameters" during vehicle operation. Data for these parameters can be collected using the vehicle's existing control modules and sensors. Additionally, temperature... and humidity It can also be obtained directly without the need for additional hardware installation. It can be represented by the following formula:

[0054] ;

[0055] On the other hand, the aforementioned visual features refer to behavioral characteristics related to motion sickness extracted from image sequences captured by in-vehicle OMS cameras. These features focus on signals that passengers (especially children with limited language abilities) cannot express verbally but will naturally convey through their bodies when experiencing motion sickness. Specifically, these include, but are not limited to: facial micro-expression features (such as a pained expression indicating furrowed brows, downturned corners of the mouth, and drooping eyelids), macro-facial color change features (such as paleness or flushing due to nausea), and head and limb movement features (such as weak backward tilting or hunching of the head due to dizziness, or covering the mouth to relieve discomfort). The process of acquiring visual features can involve analyzing images through a series of image processing algorithms, ultimately outputting a digital vector containing the aforementioned key information, thereby transforming the difficult-to-quantify discomfort manifestations into quantitative data that computers can understand and process.

[0056] It should be understood that the processes for acquiring the above-mentioned vehicle environmental features and visual features can be performed simultaneously, or sequentially, if permitted by the embodiments.

[0057] Step S220: Construct a main feature vector based on visual features; and construct an auxiliary feature vector based on visual features and vehicle environment features.

[0058] In this step, the original visual features and environmental features are mapped to a new semantic space tailored for subsequent fusion tasks through a structured processing flow (e.g., executed by a model with a specific internal architecture), and in the process, different functional roles are explicitly assigned to them.

[0059] Specifically, the acquired, pre-extracted visual features can first undergo independent feature projection or encoding. This process (which can be understood as using a fully connected layer or a small neural network) refines and transforms the visual features, converting them from a general image feature vector into a more suitable, information-density master feature vector that better represents the current core state of the passenger. In the specific task of motion sickness detection, the passenger's own physiological and behavioral manifestations are the most direct and reliable evidence of whether they are experiencing motion sickness. A frown or a gesture of covering the mouth conveys far more information than a simple vehicle bump. Therefore, the visual features can be isolated to construct a pure master feature vector unaffected by other information. This vector is designed to serve as the benchmark for information fusion in subsequent processes.

[0060] Simultaneously, the system performs another parallel processing step: combining the same visual feature with environmental features (such as acceleration and temperature) obtained from vehicle sensors, for example, through vector concatenation or element-wise addition, to form a hybrid feature that includes the user's own state and the external environmental context. This hybrid feature is then subjected to another round of independent feature projection or encoding processing to output an auxiliary feature vector. This vector is defined as a rich information base or context carrier, internally encoding joint information about "what visual state the passenger exhibits under what environmental conditions." For example, the same visual feature of "frowning" might mean "possible discomfort due to vehicle swaying" when combined with the environmental feature of "severe lateral acceleration," while when combined with the environmental feature of "stable vehicle," it might simply mean "glare from sunlight."

[0061] Through this series of parallel technical operations, the system successfully placed the same visual features into two different processing paths and output two feature vectors that are completely different in mathematical form and semantic role, laying a crucial data foundation for the next step of intelligent information fusion with a distinction between primary and secondary features.

[0062] Step S230: Based on the correlation between the main feature vector and the auxiliary feature vector, the auxiliary feature vectors are dynamically weighted and fused to generate a multimodal feature vector.

[0063] Specifically, the system uses the main feature vector (representing the passenger's core visual state) as a benchmark to calculate the degree of correlation between it and the individual elements or the whole of the auxiliary feature vector. This correlation is a mathematically defined quantitative metric used to measure the degree of matching or relevance between the current passenger's visual state and a certain piece of contextual information (such as a specific acceleration value, or its own state at another moment). The higher the correlation, the more important and relevant that piece of contextual information is for explaining the current visual state.

[0064] Next, the calculated correlation scores can be converted into specific fusion weights. The weights are directly proportional to the strength of the correlation. The system then uses these dynamically generated weights to perform a weighted summation of the auxiliary feature vectors. Thus, during the fusion process, all auxiliary information is not treated equally; instead, information highly relevant to the current visual state (i.e., with high correlation) is given greater weight in the final decision, while irrelevant information is suppressed. Finally, through this weighted summation, a new, fused multimodal feature vector is generated. This new vector is no longer a simple stack of visual and environmental information, but a high-level feature representation that has been intelligently refined and condensed, incorporating both the passenger's core state and the most relevant contextual evidence.

[0065] More specifically, in one embodiment, the process of generating the multimodal feature vector can be as follows: calculate the cosine similarity between the main feature vector and each sub-vector component of the auxiliary feature vector, and normalize it so that the sum of the weights of all components is 1. This normalized result is the final dynamic fusion weight. The higher the weight, the more relevant and consistent the state of the corresponding auxiliary feature sub-vector is with the main feature vector. Finally, the system performs a weighted summation operation, that is, multiplying each auxiliary feature sub-vector by its corresponding dynamic weight, and then adding all the weighted results to generate the final multimodal feature vector. Alternatively, in another embodiment, one or more correlation thresholds can be preset, and each calculated correlation score can be compared with the threshold. For auxiliary feature components with a correlation higher than the threshold, a fixed high weight (e.g., 1 or a value close to 1) is assigned to them; for components with a correlation lower than the threshold, a fixed low weight (e.g., 0 or a value close to 0) is assigned to them. After this gating operation, the auxiliary feature vector that finally participates in the fusion is actually composed of those high-weight components, and the contribution of the low-weight components is reduced to almost zero. Alternatively, a lightweight neural network (weight generation network) can be used to directly infer and generate a set of appropriate dynamic fusion weights for the auxiliary feature vectors, taking the main feature vector as input. Finally, the auxiliary feature vectors are weighted and summed using this set of weights generated by the network to generate the final multimodal feature vector.

[0066] Step S240: Perform motion sickness recognition processing on the multimodal feature vector to generate motion sickness recognition results for the target passenger.

[0067] The system takes the multimodal feature vector generated in the previous step as input. This vector is a high-dimensional, dense numerical representation that non-uniformly encodes the calibrated core state of the passenger and the most relevant contextual evidence. Subsequently, the system processes this vector through a neural network model that has learned the complex mapping relationship from such fused features to motion sickness state during the training phase, such as a pre-trained classifier, and outputs the motion sickness recognition result.

[0068] The final motion sickness identification result can be presented in different forms. For example, it can be a binary judgment, namely "motion sickness" or "no motion sickness". Furthermore, it can be a probability value, such as "motion sickness probability is 85%", providing a more refined basis for subsequent warnings or interventions. Alternatively, it can be a motion sickness level (such as mild, moderate, severe), thereby achieving a more accurate state assessment.

[0069] Through steps S210 to S240, an asymmetric dual-path fusion architecture, primarily driven by vision and assisted by the environment, is constructed. A dynamic weighted fusion mechanism based on correlation is introduced, which proactively filters key evidence most relevant to the current motion sickness state from multimodal information (e.g., dynamically associating a "frowning" expression with the acceleration of "severe jolts"), while automatically suppressing irrelevant or interfering information (such as similar actions caused by normal play). This generates a highly discriminative fusion feature, enabling the system to perform complete detection in the early stages before motion sickness symptoms fully manifest (e.g., vomiting, violent crying), relying solely on existing in-vehicle equipment without requiring any additional hardware. This allows for timely and accurate capture of subtle, progressive signs of discomfort, achieving high-precision, early detection, and strong anti-interference intelligent recognition of motion sickness in vehicle occupants (especially young children). This fundamentally avoids the "delayed detection of motion sickness" caused by children's insufficient language ability and parental distraction, thus effectively solving the problem of low accuracy in motion sickness recognition.

[0070] In some embodiments, the acquisition of the visual characteristics of a target occupant inside the vehicle may further include the following steps:

[0071] Image data containing the target occupants inside the vehicle is acquired. The image data is then input into a trained visual feature extraction network, which extracts fine-grained features related to motion sickness from the image data. The image data is then processed sequentially through at least one processing stage component in the visual feature extraction network, consisting of alternating basic convolutional modules and feature enhancement modules, to obtain multi-level fine-grained features from the image data. Specifically, the basic convolutional modules extract low-level visual features from the image data, while the feature enhancement module extracts multi-level fine-grained features from the low-level visual features by fusing variable convolutional kernels and channel separation strategies. Finally, the multi-level fine-grained features output from the last processing stage component in the visual feature extraction network are processed by global pooling and fully connected layers to generate visual features.

[0072] This step technically implements a hierarchical visual feature extraction process specifically designed for in-vehicle motion sickness recognition tasks. The process begins by acquiring raw image data containing the target occupant (such as images of car windows) and inputting this data into a specially trained visual feature extraction network (optimized on a dataset containing a large number of in-vehicle scenes and motion sickness-related expressions and postures). The core architecture of this network consists of a series of processing stage components, alternating between at least one basic convolutional module (Conv component) and a feature enhancement module (C3k2 component), and a fully connected layer (FC) is added at the end of the network to complete the final integration and output of features.

[0073] Please see Figure 3The basic convolution module (Conv component) serves as the front end of each processing stage, containing three key steps: single convolution operation, batch normalization processing, and activation function operation. It can efficiently extract the low-level visual features of the image (such as edges, textures, etc.). The feature enhancement module (C3k2 component), a signature feature extraction enhancement module in the YOLO11 model, is an optimized design based on the traditional C3 module. This module performs deep processing on the low-level features output by the basic convolutional module. By fusing variable convolutional kernel technology (i.e., using convolutional kernels with different receptive fields in parallel or adaptively to capture multi-scale visual patterns, such as simultaneously capturing subtle facial wrinkles and macroscopic head and neck postures) and channel separation strategy (splitting the feature flow into independently processed branches to enhance feature expressiveness and alleviate gradient vanishing), it is specifically designed to further extract multi-level, fine-grained features that are highly relevant to motion sickness discrimination and have more semantic information (e.g., local features describing the degree of eyelid drooping, regional features reflecting changes in facial color, and global features representing the degree of body curling) from low-level features. This significantly improves the ability to capture multi-level, fine-grained features in complex scenes, especially suitable for deep feature extraction tasks in scenes with complex lighting changes and diverse target postures, such as inside a car.

[0074] By organically combining the Conv component for basic feature extraction with the C3k2 component for deep feature enhancement, this alternating stacked structure allows the network to abstract and enhance features layer by layer and progressively. This feature extraction module balances efficiency and depth, preserving image details while capturing high-order semantic features related to motion sickness, thus providing more comprehensive and discriminative feature support for subsequent motion sickness recognition tasks. Finally, the last-stage processing component outputs a highly refined feature map that incorporates multi-scale contextual information.

[0075] Subsequently, the feature map is processed by a global pooling layer (usually global average pooling) to eliminate spatial dimensionality, compressing it into a fixed-length feature vector while preserving its most significant channel responses and enhancing the model's translation invariance. This vector is then fed into a fully connected layer for final linear / nonlinear transformation and dimensionality reduction, aggregating and mapping the rich visual information extracted by all previous modules into a compact and highly discriminative visual feature vector. This provides a direct and robust visual representation for subsequent multimodal fusion and motion sickness recognition.

[0076] It should also be noted that the construction of the training set for the aforementioned visual feature extraction network can be carried out as follows: Collect diverse image data of children of different ages in car interiors (covering different sitting postures, lighting conditions, seat types, etc.), and then use professional annotation tools to accurately annotate the child targets in the images to form standardized training data; subsequently, the YOLO11 model is specifically trained based on this dataset to learn and master the visual feature patterns of children in cars; after training, the in-car images captured in real time by the vehicle camera are input into the model, and the model can quickly output accurate detection boxes of child targets, achieving effective identification and localization of children in cars.

[0077] In some embodiments, the construction of the auxiliary feature vector based on visual features and vehicle environment features may further include the following steps:

[0078] Obtain the voice characteristics of the target occupant; construct an auxiliary feature vector based on visual characteristics, vehicle environment characteristics, and voice characteristics.

[0079] A child's voice (such as crying) can reflect a child's motion sickness to some extent, but since it is not as intuitive as a visual feature and is prone to error, it is used as an auxiliary feature in this embodiment.

[0080] The system can capture the raw audio stream inside the vehicle using an in-vehicle microphone array and extract the sound signals belonging to the target occupant (such as crying, sobbing, groaning, etc.). This audio data is then processed; for example, log-Mel spectrograms can be used for audio processing. In the feature extraction network design, the network structure used to extract log-Mel spectrogram features can also be... Figure 3 Similar overlapping network structures are used for feature extraction, and the extracted sound feature vector is denoted as... .

[0081] This sound feature vector is combined with the previously acquired visual feature vector and vehicle environment feature vector through multimodal feature processing (usually by vector concatenation or element-wise addition) to form a unified hybrid feature representation that integrates the occupant's own behavior, vehicle motion state, and acoustic environment. Finally, an auxiliary feature vector rich in multidimensional contextual information is output for subsequent attention fusion.

[0082] Through the above embodiments, by introducing sound features, it is beneficial to solve the problem of misjudgment and missed judgment caused by information loss or conflict in complex real driving scenarios of single or dual-modal perception. It significantly enhances the system's comprehensive judgment ability and robustness in complex scenarios such as visual information being occluded or environmental interference.

[0083] In some embodiments, the above-described motion sickness recognition processing of multimodal feature vectors to generate motion sickness recognition results for the target occupant may further include the following steps:

[0084] Multimodal feature vectors from continuous time series are sequentially input into the trained temporal feature extraction model. The temporal feature extraction model is used to capture the dynamic changes of multimodal feature vectors in continuous time series, and motion sickness recognition results are generated based on the dynamic changes.

[0085] This involves concatenating individual multimodal feature vectors from multiple time points in chronological order to form a sequence, and then performing in-depth analysis on the input continuous multimodal feature vector sequence using a pre-trained temporal feature extraction model (such as a recurrent neural network (RNN) or a long short-term memory network (LSTM)).

[0086] This temporal feature extraction model relies on its internal recurrent connections or causal convolutional structures to receive a multimodal feature vector at each time step. It records and transmits historical information through its hidden states, thus constructing a contextual memory of the entire temporal sequence. Based on this structure, the model uses complex internal nonlinear transformations to progressively uncover the hidden dynamic patterns within the sequence. Finally, the model encodes these learned dynamic patterns into a comprehensive temporal context vector and maps it to motion sickness recognition results through a terminal classifier (such as a fully connected layer and Softmax). This allows the system to make judgments based on the continuous evolution of symptoms rather than isolated moments, enabling it to keenly capture early signs of motion sickness, robustly filter transient disturbances, and accurately assess the development stage of motion sickness. This fundamentally solves the problems of misjudgment and lag caused by the lack of temporal context in single-frame analysis.

[0087] In related technologies, traditional algorithms such as RNN / LSTM / GRU lack parallel computing capabilities, resulting in extremely low training efficiency. Furthermore, when the sequence length is too large, their gradients decay significantly, leading to the ineffective transmission of long-distance contextual information. Since motion sickness is a dynamic process, using only a short period of data (such as 25 frames per second) may prevent the model from learning effective information effectively; therefore, RNN-based algorithms are not suitable.

[0088] Based on this, in some embodiments, the multimodal feature vectors on the continuous time series include: the multimodal feature vector of the current frame and the multimodal feature vectors of historical frames; the step of sequentially inputting the multimodal feature vectors on the continuous time series into the trained temporal feature extraction model for temporal processing to generate motion sickness recognition results may further include the following steps:

[0089] The multimodal feature vectors from the continuous time series are sequentially input into the temporal feature extraction model. Through the first attention layer in the temporal feature extraction model, the multimodal feature vector of the current frame is used as the first query vector, and the multimodal feature vectors of the historical frames are used as the first key value vector. Based on the first query vector, cross-attention calculation is performed on the first key value vector to output the motion sickness recognition result.

[0090] Specifically, the system distinguishes the multimodal feature vectors on a continuous time series into two functionally defined sets: the current frame multimodal feature vector representing the latest state and the historical frame multimodal feature vector carrying historical information.

[0091] When the time-series feature extraction model processes this sequence, its internal first attention layer constructs an asymmetric attention computation architecture, using the feature vector of the current frame as the first query vector. Simultaneously, it uses the feature vectors of all historical frames as the first key-value vector. The key vectors provide index information for matching the query, while the value vectors store specific state details, together forming a complete historical memory. Next, the system performs cross-attention computation, calculating the similarity score between the query vector and all key vectors through matrix operations. After Softmax normalization, a set of attention weights is generated. This set of weights dynamically reflects the correlation between the state at each moment in the historical sequence and the current state. Finally, the value vectors are weighted and summed using these weights to output a context vector that incorporates all relevant historical evidence. This vector is essentially a current state representation enhanced with historical information. It retains the immediacy of the current frame while incorporating dynamic patterns in historical frames that form causal, progressive, or contrastive relationships with the current state (such as the gradual intensification of motion sickness symptoms). Ultimately, motion sickness recognition results are generated based on this deeply optimized temporal feature, thereby achieving accurate capture and early warning of the dynamic process of motion sickness.

[0092] In some embodiments, the first attention layer is a deformable attention layer; the above-mentioned calculation of cross-attention on the first key-value vector based on the first query vector to output the motion sickness recognition result may further include the following steps:

[0093] The sampling offset of the first query vector is dynamically learned through a deformable attention layer. The key positions in the first key value vector are extracted based on the sampling offset. The key positions are weighted and fused based on the first query vector to output the motion sickness recognition result.

[0094] In this step, a deformable attention layer optimizes traditional attention computation. Specifically, this layer takes the first query vector (representing the enhanced state of the current frame) output by the previous attention layer as input and dynamically predicts a set of sampling offsets through a lightweight regression network. These offsets do not directly affect the query vector itself, but rather indicate the key position indices that need to be focused on in the first key-value vector (i.e., the historical frame feature sequence). The system then sparsely samples a few of the most critical historical frame feature blocks from the originally lengthy historical sequence based on these learned offsets, thereby significantly compressing the data scale to be processed from the entire sequence to only a few key moments most relevant to the current state. Next, the system uses the first query vector as a benchmark and performs simplified attention interactions only with these selected key positions, calculates their correlation, generates corresponding fusion weights, and finally generates motion sickness recognition results by weighted summation of these key features. This mechanism not only significantly reduces the computational overhead of long sequence processing, but also effectively captures the most discriminative dynamic pattern features during motion sickness by actively focusing on critical frames of symptom evolution (such as the key moments when facial expressions begin to change abruptly or action patterns switch), thereby achieving earlier and more accurate identification and warning while ensuring the real-time performance of the model.

[0095] In some embodiments, the construction of the main feature vector based on visual features and the construction of the auxiliary feature vector based on visual features and vehicle environment features may further include the following steps:

[0096] Visual features and vehicle environment features are input into the trained multimodal fusion model. The multimodal fusion model includes at least a main branch and an auxiliary branch. The visual features are processed by feature projection through the main branch of the multimodal fusion model to output the main feature vector. The visual features and vehicle environment features are processed by feature projection through the auxiliary branch of the multimodal fusion model to output the auxiliary feature vector.

[0097] The feature vector is constructed using a multimodal fusion model with a clearly defined division of labor. This model's architecture includes at least a main branch and auxiliary branches for parallel processing. When the system inputs the extracted visual features and vehicle environment features into the model, the internal processing is as follows: On the main branch, the model uses one or more fully connected layers to perform feature projection processing on the input visual features. This process maps the original visual features to a new feature space through a combination of linear transformations and nonlinear transformations, ultimately outputting a main feature vector, which serves as the benchmark for subsequent fusion processes. Simultaneously, on the auxiliary branch, the model first concatenates or fuses the same set of visual features with the vehicle environment features to form a joint feature that includes both its own state and the external environment. This joint feature is then encoded and transformed using another independent set of feature projection processes (such as fully connected layers), aiming to learn and integrate the correlation between visual and environmental features, ultimately outputting an auxiliary feature vector that acts as a contextual information base.

[0098] It should also be understood that when introducing sound features, the same method can be used to input the sound features into the model for fusion. Please refer to [link / reference] for details. Figure 4 Among them, visual feature X m Vehicle environmental characteristics X s With sound feature X a The input is fed into a multimodal fusion model, and after feature extraction through an FC layer, one output is the main feature vector, and the other output is the visual feature X. m Vehicle environmental characteristics X s Sound characteristics X a After being combined, the data are fed into the encoder for further encoding as an auxiliary branch.

[0099] Through the above embodiments, the dominant role of visual information in the decision-making process is forcibly established through structured model design. At the same time, rich contextual representations are constructed using environmental information, thereby achieving a clear distinction between primary and secondary information fusion at the algorithm level. This fundamentally avoids the problems of blurred feature weights and mutual interference between modalities that are common in traditional multimodal fusion.

[0100] In some embodiments, the above-mentioned multimodal fusion model further includes a second attention layer; based on the main feature vector, fusing auxiliary feature vectors to generate a multimodal feature vector may further include the following steps:

[0101] The second attention layer uses the main feature vector as the second query vector and the auxiliary feature vector as the second key vector. Based on the second query vector, attention is calculated on the second key vector to output a multimodal feature vector.

[0102] Specifically, the system inputs the main feature vector obtained from the main branch processing as the second query vector into the second attention layer; simultaneously, it uses the auxiliary feature vector output from the auxiliary branch as the second key-value vector. The key vectors form a searchable index feature library, while the value vectors store the corresponding visual-environment joint feature details. The system then executes the attention calculation process: first, it calculates the dot product similarity between the query vector and all key vectors through matrix operations; then, it normalizes this to a probability distribution form of attention weights using the Softmax function. This process essentially dynamically evaluates the importance of each context element in the auxiliary feature vector based on the core state represented by the main feature vector. Finally, these weights are used to perform a weighted summation of the value vectors. The resulting multimodal feature vector retains the dominant information of the main feature vector while selectively incorporating the most relevant environmental context features, thus achieving a sublimation from feature stacking to information optimization fusion, providing highly discriminative feature representations for subsequent recognition tasks.

[0103] Please continue reading. Figure 4 Visual, environmental, and sound features are extracted into X. m With X s X a Afterwards, one route was X m The output feature vector serves as the main branch and also as the key Q in the attention mechanism. Another route is X. m With X s X a The combined features are fed into the encoder for further encoding as an auxiliary branch, and also serve as key and value (K and V). The features after passing through the attention layer are then fed into the fully connected (FC) layer for prediction. The FC layer is a fully connected layer, and the attention layer is an attention layer. The final output of the model after passing through the FC fully connected layer is a feature vector extracted from a single frame image and environmental information.

[0104] The present application will now be described and illustrated with reference to specific embodiments. Figure 5 This is a flowchart of another motion sickness recognition method according to an embodiment of this application, such as... Figure 5 As shown, the process includes the following steps:

[0105] Step S501: Identify the visual characteristics of a child inside the vehicle. This involves first using an OMS camera to identify image data, and then using YOLO11 to identify the child's recognition area.

[0106] Step S502: Environmental and sound feature processing. To avoid misjudgments (such as deviations in facial expressions or behavior caused by children playing normally or changes in lighting inside the vehicle), the algorithm incorporates environmental auxiliary data collected by the OMS camera for calibration.

[0107] Step S503, cross-modal feature fusion; thereby realizing a cross-modal fusion method with visual features as the main component and environmental features as the auxiliary component.

[0108] Step S504, Temporal Feature Prediction. The fused features are input into the temporal feature prediction model, and the final output is the prediction result.

[0109] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0110] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0111] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0112] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0113] S1, acquire vehicle environmental features and visual features of target occupants inside the vehicle; vehicle environmental features include vehicle state parameters and environmental parameters associated with inducing motion sickness in passengers inside the vehicle, and visual features are behavioral characteristics related to motion sickness.

[0114] S2, construct the main feature vector based on visual features; and construct the auxiliary feature vector based on visual features and vehicle environment features.

[0115] S3, based on the correlation between the main feature vector and the auxiliary feature vector, dynamically weights and fuses each auxiliary feature vector to generate a multimodal feature vector.

[0116] S4 performs motion sickness recognition processing on the multimodal feature vectors to generate motion sickness recognition results for the target passenger.

[0117] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0118] In addition, in conjunction with the motion sickness recognition methods in the above embodiments, this application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the motion sickness recognition methods in the above embodiments.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0120] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0121] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A motion sickness detection method, characterized in that, The method includes: Acquire vehicle environmental features and visual features of the target occupants inside the vehicle; the vehicle environmental features include vehicle state parameters and environmental parameters associated with inducing motion sickness in passengers inside the vehicle, and the visual features are behavioral characteristics related to motion sickness. A main feature vector is constructed based on the visual features; and an auxiliary feature vector is constructed based on the visual features and the vehicle environment features. Based on the correlation between the main feature vector and the auxiliary feature vector, the auxiliary feature vectors are dynamically weighted and fused to generate a multimodal feature vector; The multimodal feature vector is processed for motion sickness recognition to generate motion sickness recognition results for the target passenger.

2. The motion sickness recognition method according to claim 1, characterized in that, The step of performing motion sickness recognition processing on the multimodal feature vectors to generate motion sickness recognition results for the target passenger includes: The multimodal feature vectors from the continuous time series are sequentially input into the trained time series feature extraction model; The time-series feature extraction model is used to capture the dynamic change pattern of the multimodal feature vector in the continuous time series, and the motion sickness recognition result is generated based on the dynamic change pattern.

3. The motion sickness recognition method according to claim 2, characterized in that, The multimodal feature vectors on the continuous time series include: the multimodal feature vector of the current frame and the multimodal feature vectors of historical frames; The step of sequentially inputting multimodal feature vectors from a continuous time series into a trained temporal feature extraction model for temporal processing to generate the motion sickness recognition result includes: The multimodal feature vectors on the continuous time series are sequentially input into the temporal feature extraction model. Through the first attention layer in the temporal feature extraction model, the multimodal feature vector of the current frame is used as the first query vector, and the multimodal feature vector of the historical frames is used as the first key value vector. Based on the first query vector, cross-attention calculation is performed on the first key value vector to output the motion sickness recognition result.

4. The motion sickness recognition method according to claim 3, characterized in that, The first attention layer is a deformable attention layer; The step of performing cross-attention calculation on the first key-value vector based on the first query vector and outputting the motion sickness recognition result includes: The sampling offset of the first query vector is dynamically learned through the deformable attention layer, and the key positions in the first key-value vector are extracted based on the sampling offset. The motion sickness recognition result is output by weighting and fusing the key positions based on the first query vector.

5. The motion sickness recognition method according to claim 1, characterized in that, The main feature vector is constructed based on the visual features; And, based on the visual features and the vehicle environment features, an auxiliary feature vector is constructed, including: The visual features and the vehicle environment features are input into the trained multimodal fusion model; the multimodal fusion model includes at least a main branch and an auxiliary branch; The visual features are processed by feature projection through the main branch of the multimodal fusion model to output the main feature vector, and the visual features and the vehicle environment features are processed by feature projection through the auxiliary branch of the multimodal fusion model to output the auxiliary feature vector.

6. The motion sickness recognition method according to claim 5, characterized in that, The multimodal fusion model further includes a second attention layer; the step of fusing the auxiliary feature vectors based on the main feature vector to generate a multimodal feature vector includes: The main feature vector is used as the second query vector and the auxiliary feature vector is used as the second key vector via the second attention layer. Attention is calculated on the second key vector based on the second query vector to output the multimodal feature vector.

7. The motion sickness recognition method according to claim 1, characterized in that, The step of constructing an auxiliary feature vector based on the visual features and the vehicle environment features includes: Acquire the voice characteristics of the target occupant; The auxiliary feature vector is constructed based on the visual features, the vehicle environment features, and the sound features.

8. The motion sickness recognition method according to any one of claims 1 to 7, characterized in that, Acquiring the visual characteristics of the target occupant inside the vehicle, including: Acquire image data containing the target occupants inside the vehicle; The image data is input into a trained visual feature extraction network, which is used to extract fine-grained features related to motion sickness from the image data. The image data is processed sequentially through at least one processing stage component in the visual feature extraction network, which consists of alternating basic convolutional modules and feature enhancement modules, to obtain multi-level fine-grained features. The basic convolutional module is used to extract low-level visual features from the image data, and the feature enhancement module is used to extract multi-level fine-grained features from the low-level visual features by fusing variable convolutional kernels and channel separation strategies. The multi-level fine-grained features output by the last-stage processing component in the visual feature extraction network are processed by global pooling and fully connected layers to generate the visual features.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the motion sickness recognition method according to any one of claims 1 to 8.

10. A vehicle, characterized in that, It includes the vehicle body and the electronic device as described in claim 9.

Images