Passenger state adjustment method, vehicle, and storage medium

Abstract

The present application relates to a kind of passenger state regulation method, carrier and storage medium.The method includes: by sensing component, the multi-modal physiological signal of passenger is collected;Based on the failure probability of the sensing component, the time characteristics and spatial characteristics extracted in the multi-modal physiological signal are fused, and the fusion feature vector is obtained;According to the fusion feature vector, the passenger state type is determined, and the state regulation operation corresponding to the passenger state type is executed.The present application improves the classification accuracy and robustness of passenger state type, and then improves the regulation effect of passenger state and passenger experience.

Description

Technical Field

[0001] This invention relates to the field of passenger status adjustment technology, specifically to a passenger status adjustment method, vehicle, and storage medium. Background Technology

[0002] Currently, passenger comfort and safety are paramount in urban transportation vehicles (such as aircraft), requiring real-time monitoring and timely adjustments to abnormal conditions. Existing technologies can determine passenger abnormalities based on physiological signals; however, these processes do not comprehensively consider the temporal correlation (the varying importance of physiological signals at different times) and spatial correlation (such as the coupling relationships between different physiological signals). This leads to information loss, making accurate judgments about passenger conditions impossible, and consequently, inaccurate adjustments, significantly impacting the passenger experience. Summary of the Invention

[0003] To address the problem of inaccurate passenger status judgment and adjustment in existing technologies, embodiments of the present invention provide a passenger status adjustment method, a vehicle, and a storage medium.

[0004] This invention provides a passenger status adjustment method, including: Passengers' multimodal physiological signals are collected through sensing components; Based on the failure probability of the sensing component, the temporal and spatial features extracted from the multimodal physiological signal are fused to obtain a fused feature vector. The passenger state type is determined based on the fused feature vector, and a state adjustment operation corresponding to the passenger state type is performed.

[0005] This invention also provides a vehicle including the aforementioned controller, the controller including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the passenger state adjustment method when executing the computer-readable instructions.

[0006] This invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the passenger state adjustment method.

[0007] The present invention provides a passenger state adjustment method, vehicle, and storage medium. The method includes: acquiring multimodal physiological signals of passengers through a sensing component; fusing temporal and spatial features extracted from the multimodal physiological signals based on the failure probability of the sensing component to obtain a fused feature vector; determining the passenger state type according to the fused feature vector, and performing a state adjustment operation corresponding to the passenger state type.

[0008] In this invention, after acquiring multimodal physiological signals through sensing components, the temporal and spatial features extracted from the multimodal physiological signals are fused based on the failure probability of the sensing components to obtain a fused feature vector. During the fusion of the fused feature vector, the temporal correlation of each physiological signal in the multimodal physiological signals (i.e., the importance of the same physiological signal varies at different time points) and the spatial correlation between different physiological signals (i.e., the coupling relationship between different physiological signals) are fully considered based on the temporal features. Therefore, the passenger state type determined by the fused feature vector comprehensively considers spatiotemporal characteristics, improving the accuracy of passenger state type classification. Simultaneously, since the failure probability of the sensing components is also considered during the fusion of the fused feature vector, the robustness of passenger state type classification is also improved. Consequently, with improved accuracy and robustness in passenger state type classification, the final state adjustment operation performed based on the passenger state type will be more closely matched to the passenger's current actual state, improving the adjustment effect and passenger experience. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a passenger status adjustment method according to an embodiment of the present invention; Figure 2 This is a flowchart of step S10 of the passenger state adjustment method in one embodiment of the present invention; Figure 3 This is a flowchart of step S20 of the passenger state adjustment method in one embodiment of the present invention; Figure 4 This is a flowchart of a passenger state adjustment method in another embodiment of the invention; Figure 5 This is a schematic diagram of a controller in one embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In one embodiment, such as Figure 1 As shown, a passenger status adjustment method is provided, including the following steps S10-S30: S10, collecting multimodal physiological signals from passengers via sensing components; wherein, the vehicle includes aircraft (such as electric vertical takeoff and landing aircraft), automobiles, high-speed trains, and other means of transportation for passenger use. The sensing component refers to a miniature, non-invasive biosensor array deployed within the vehicle for passenger use in the seat or surrounding environment, comprising multiple sensors. Specifically, the sensors in the sensing component can be deployed in areas that directly contact the passenger, such as multiple sensors distributed across the headrest, armrests, footrests, etc. Some sensors can be integrated into smart wearable devices such as smartwatches or smart helmets to ensure signal acquisition quality through direct contact with the passenger while avoiding impact on passenger comfort; however, sensors can also be deployed in locations around the seat that do not directly contact the passenger, such as cameras, radar, etc. Sensors capable of non-contact monitoring can be placed opposite the seat. The sensors must maintain signal quality even after deformation and must cover all physiological signals in the multimodal physiological signal range to ensure the integrity of signal acquisition. The multimodal physiological signals include at least two of the following: electrocardiogram (ECG) signals, skin conductance signals, respiratory signals, body temperature signals, blood oxygen saturation signals, electroencephalogram (EEG) signals, and eye-tracking signals. The sensing components may include at least two of the following: ECG sensors, skin conductance sensors, respiratory rate sensors, body temperature sensors, blood oxygen saturation sensors, EEG sensors, and eye-tracking sensors, to facilitate the corresponding acquisition of each physiological signal within the multimodal physiological signals.

[0013] In one embodiment, such as Figure 2 As shown, step S10, namely the acquisition of passengers' multimodal physiological signals through sensing components, includes the following steps S101-S103: S101, acquire the signal fluctuation frequency range of each physiological signal to be collected, and determine the acquisition frequency of each physiological signal according to the signal fluctuation frequency range; wherein, the acquisition frequency of each physiological signal can be determined according to the signal fluctuation frequency range of that physiological signal. For example, the acquisition frequency of the physiological signal is at least twice the highest frequency in the signal fluctuation frequency range. For example, the signal fluctuation frequency range of the electrocardiogram signal is 0.5-40Hz, and its corresponding acquisition frequency can be set to 250Hz; while the signal fluctuation frequency range of the skin conductance activity signal is 0-10Hz, and its corresponding acquisition frequency can be set to 50Hz; the signal fluctuation frequency range of the respiratory rate is 0.1-0.5Hz, and its corresponding acquisition frequency can be set to 20Hz; the body temperature changes slowly, and the maximum frequency of its corresponding signal fluctuation frequency range is much less than 1Hz, so its corresponding acquisition frequency can be set to 1Hz.

[0014] S102, the sensing components acquire the physiological signals according to the respective acquisition frequencies. Specifically, the sensing components may include electrocardiogram (ECG) sensors, skin conductance sensors, respiratory rate sensors, body temperature sensors, blood oxygen saturation sensors, electroencephalogram (EEG) sensors, eye-tracking sensors, etc. In some embodiments, the multimodal physiological signals in this invention include ECG signals, skin conductance signals, respiratory signals, and body temperature signals. In this case, the following sensors can be configured in the sensing components: The electrocardiogram (ECG) signal sensor includes a main ECG sensor and a backup ECG sensor. The main ECG sensor is integrated into the seat back, employing dry electrode technology with a sampling frequency of 250Hz. The backup ECG sensor is integrated into the seat armrest, employing photoplethysmography (PPG) technology with a sampling frequency of 100Hz. Since ECG signals are crucial indicators for assessing a passenger's cardiovascular status, the main and backup ECG sensors utilize different technological principles to ensure that the other can continue functioning even if one technology fails. In a further embodiment, photoplethysmography (PPG) signals can be used to detect changes in vascular volume using green or infrared light, combined with an inertial measurement unit (IMU) to capture motion noise at the wearing site, determining the peripheral vascular volume signal of the heartbeat, and replacing the ECG signal in the multimodal physiological signal analysis.

[0015] The electroskin activity (ESA) sensor includes a primary ESA sensor and a backup ESA sensor. The primary ESA sensor is integrated into the seat cushion and uses a dual-electrode measurement method with a sampling frequency of 50 Hz. The backup ESA sensor is integrated into the seat back and uses a single-electrode measurement method with a sampling frequency of 50 Hz. Since ESA signals reflect a passenger's stress and emotional state, the dual-sensor deployment of the primary and backup ESA sensors improves data reliability. In a further embodiment, an infrared sensor can also be used for infrared thermal imaging to identify emotional signals, replacing the ESA signal in the multimodal physiological signal analysis.

[0016] The respiratory rate sensor includes a main respiratory rate sensor and a backup respiratory rate sensor. The main respiratory rate sensor is integrated into the seat back and uses a piezoresistive sensor with a sampling frequency of 20Hz. The backup respiratory rate sensor is integrated into the seat belt and uses an accelerometer for indirect measurement with a sampling frequency of 50Hz. Since respiratory signals are an important indicator for assessing passenger comfort, the main and backup respiratory rate sensors employ different measurement principles to ensure that the other can still function if one technology fails. In some embodiments, millimeter-wave radar can also be used for non-contact monitoring to replace the respiratory rate sensor in collecting respiratory signals.

[0017] The body temperature sensor includes a main body temperature sensor and a backup body temperature sensor. The main body temperature sensor is integrated into the seat cushion and uses an infrared temperature sensor with a sampling frequency of 1Hz. The backup body temperature sensor is integrated into the seat back and uses a contact temperature sensor with a sampling frequency of 1Hz. Changes in body temperature signals reflect the passenger's physiological state, and the dual-sensor deployment improves measurement accuracy.

[0018] In some embodiments, in addition to ECG signal sensors, skin conductance sensors, respiratory rate sensors, and body temperature sensors, a blood oxygen saturation sensor can be added to expand the ability to detect hypoxia, or / and an EEG sensor can be added to expand the fatigue monitoring dimension, or / and an eye-tracking sensor can be added to expand the attention state assessment. Thus, in this embodiment, the comprehensiveness and accuracy of subsequent classification of passenger state types based on multimodal physiological signals can be improved by adding modal dimensions.

[0019] S103, after preprocessing all the collected physiological signals, multimodal physiological signals are obtained.

[0020] Understandably, after collecting all physiological signals, the raw physiological signals can be preprocessed in real time using an edge neural network processor, specifically including the following processing steps: Filtering is performed using different methods for different physiological signals. The filtering method is set according to the frequency characteristics of each physiological signal to ensure that the effective signal passes through while noise is filtered out. For example, a 0.5-40Hz bandpass filter is used to remove power frequency interference and baseline drift in the electrocardiogram signal; a 0.1-5Hz lowpass filter is used to extract the slow-changing component of skin conductance in the skin conductance signal; a 0.1-0.5Hz bandpass filter is used to extract the respiratory rate in the respiratory signal; and a moving average filter is used to filter the body temperature signal with a window size of 5 seconds.

[0021] For noise reduction, wavelet denoising is used to remove high-frequency noise and retain effective signal components.

[0022] Normalization processing normalizes the original physiological signals of different dimensions to [0, 1]. ， [1] This interval facilitates subsequent feature fusion processing. Its normalization formula is as follows: xnorm = (x - x_min) / (xmax - xmin) Where: x is the signal value of the original physiological signal, xmin is the minimum value of the original physiological signal, xmax is the maximum value of the original physiological signal, and xnorm is the normalized value of the original physiological signal.

[0023] S20, based on the failure probability of the sensing component, the temporal and spatial features extracted from the multimodal physiological signal are fused to obtain a fused feature vector; wherein, the failure probability of the sensing component can be determined based on the average fault interval time of the sensor, for example, the failure probability can be 10. -6 / hour. In some embodiments, temporal and spatial features can be fused using a spatiotemporal attention mechanism. This allows for highlighting important time points based on temporal features in the temporal dimension and important physiological signals based on spatial features in the spatial dimension, thereby achieving weighted fusion to obtain a fused feature vector. However, in this invention, feature fusion of temporal and spatial features can also be achieved using Transformer encoder architectures, graph attention networks, spatiotemporal convolutional networks, etc., to obtain a fused feature vector, as long as it can achieve the fusion processing of temporal and spatial features and ultimately realize the effective fusion of multimodal signals.

[0024] In one embodiment, such as Figure 3 As shown, step S20, which is the feature fusion of temporal and spatial features extracted from the multimodal physiological signal based on the failure probability of the sensing component to obtain a fused feature vector, includes the following steps S201-S203: S201, extract the temporal and spatial features of the multimodal physiological signals using a preset extraction model; wherein, the temporal features can be extracted using a one-dimensional convolutional neural network in the preset extraction model, wherein the kernel size of the one-dimensional convolutional neural network is 3, the stride is 1, and the activation function is ReLU. A graph convolutional network (GCN) is used to extract the spatial correlation between different physiological signals, constructing a physiological signal graph (i.e., spatial features), wherein nodes in the physiological signal graph represent different physiological signals, and edges represent the correlation between different physiological signals.

[0025] S202, determine the temporal attention weight based on all the temporal features, determine the initial spatial attention weight based on all the spatial features, and correct the initial spatial attention weight based on the failure probability and reliability index of the sensing component to obtain the target spatial attention weight; in this step, it is necessary to simulate the neuronal connection mechanism of the human cerebral cortex through a spatiotemporal attention mechanism to perform weighted fusion of temporal and spatial features in multimodal physiological signals.

[0026] First, we need to calculate the temporal attention weight (which time point's physiological signal is more important to focus on) and the target spatial attention weight (which physiological signal is more important to focus on). The formula for calculating the temporal attention weight is as follows: at （i） =exp(et （i） ) / Σ j exp(e_t （j） ) e_t （i） =W_t T tanh(W_hh_t （i） +b_t) in: at （i） The temporal attention weight for the i-th time step is used to characterize the importance of the physiological signal at that time point during fusion, and the sum of the temporal attention weights for all time steps equals 1.

[0027] et （i） The attention score for the i-th time step is the median value used to calculate the time attention weight. The higher the attention score, the more attention is paid to the physiological signals at that time step.

[0028] exp(et （i） () refers to the attention score et at the i-th time step. （i） The results of the exponential calculation are used to amplify the differences in scores and strengthen the weighting of important time steps.

[0029] Σ j exp(e_t （j）) refers to the summation of the exponential attention scores over all time steps (j represents each time step).

[0030] h_t （i） It is the feature vector of the i-th time step; that is, the global feature vector after integrating the time features corresponding to the physiological signals collected by multiple sensors at time t.

[0031] W_t is the first learnable parameter matrix; W_t T This is the transpose of the first learnable parameter matrix; W_h is the second learnable parameter matrix.

[0032] b_t is the first learnable bias term.

[0033] The formula for calculating the attention weights in the target space is as follows: as （m） =exp(e_s （m） ) / Σ n exp(e_s （n） ) e_s （m） =W_s T tanh(W_gg_s （m） +b_s) in: as （m） Let be the initial spatial attention weight for the m-th physiological signal, representing the importance of this type of physiological signal during fusion. The sum of the weights of all types of physiological signals is 1.

[0034] e_s （m） is the importance score of the m-th physiological signal, which is also the median value used to calculate the initial spatial attention weight. The higher the importance score, the more attention is paid to this type of physiological signal.

[0035] exp(e_s （m） () refers to the importance score e_s for the m-th signal. （m） The result of the exponential operation is used to amplify the difference in scores, making the more important signal types stand out.

[0036] Σ n exp(e_s （n） ) refers to the summation of the indexed importance scores of all types of physiological signals (n represents each type of signal).

[0037] g_s （m） Let m be the feature vector of the m-th physiological signal; for example, the feature vector of an electrocardiogram (ECG) signal. W_s is the third learnable parameter matrix; W_sT This is the transpose of the third learnable parameter matrix; W_g is the fourth learnable parameter matrix; b_s is the second learnable bias term.

[0038] Understandably, in this embodiment, after determining the initial spatial attention weights, it is also necessary to consider the possibility of sensor failure. Based on the failure probability and reliability index of the sensing component, the initial spatial attention weights are corrected to obtain the target spatial attention weights, so that the fused feature vector obtained in the final fusion is more reliable.

[0039] The failure probability model used to determine the failure probability is as follows: Pfail （m，k） =1-exp(-λ{m,k}·t) in: t represents the current time; m represents the m-th physiological signal; k is the kth sensor; Pfail （m，k） The failure probability of the k-th sensor for the m-th physiological signal (each of the multimodal physiological signals is measured by two sensors in the sensing component; each physiological signal corresponds to two sensors, that is, one main sensor and one backup sensor; where the main sensor k=1 and the backup sensor k=2) at the current time t; λ{m, k} are failure parameters, determined based on historical data and sensor characteristics.

[0040] exp( . ) is an exponential function used to simulate the increasing probability of sensor failure over time.

[0041] The initial spatial attention weights are corrected using the following correction function to obtain the target spatial attention weights: a （m） =exp(es （m） )·[(1-Pfail （m，1） )·R （m，1） +(1-Pfail （m，2） )·R （m，2） ] / Σ n exp(es （n） )·[((1-Pfail （n，1） )·R （n，1） +(1-Pfail （n，2） )·R （n，2） )] in: a （m）The corrected target space attention weights are for the m-th physiological signal. exp(e_s （m） () refers to the importance score e_s for the m-th signal. （m） The result of exponentiation; Σ n exp(e_s （n） ) refers to the summation of the indexed importance scores of all types of physiological signals (n represents each type of signal).

[0042] Pfail （m，1） Let 1 - Pfail represent the failure probability of the main sensor (k=1) for the m-th physiological signal at the current time t; （m，1） It is the probability that the main sensor of the m-th physiological signal is "not malfunctioning and can work normally" at the current time t; the m-th physiological signal here represents the physiological signal that is currently being calculated (for example, if we are now calculating the target space attention weight of the electrocardiogram signal, m refers to the electrocardiogram signal).

[0043] R （m，1） The reliability index of the main sensor for the m-th physiological signal; Pfail （m，2） Let 1 - Pfail be the failure probability of the backup sensor (k=1 for the backup sensor) for the m-th physiological signal at the current time t; （m，2） It is the probability that the backup sensor for the m-th physiological signal is "not malfunctioning and can work normally" at the current time t; R （m，2） The reliability index of the backup sensor for the m-th physiological signal; Pfail （n，1） Let n be the failure probability of the main sensor (k=1 for the main sensor) for the nth physiological signal; n refers to one of the physiological signals when traversing all physiological signals.

[0044] R （n，1） The reliability index of the main sensor for the nth physiological signal; Pfail （n，2） Let f(x) be the failure probability of the backup sensor for the nth physiological signal (k=2 for the backup sensor).

[0045] R （n，2） The reliability index of the backup sensor for the nth physiological signal; The reliability index is equal to the ratio of the current signal-to-noise ratio (SNR) to the maximum SNR of the physiological signal. Let R be the reliability index of the k-th sensor for the m-th physiological signal. （m，k） For example (and others similarly), R （m，k） The formula for calculating the reliability index is: R （m，k） =SNR（m，k） / SNR_max （m） Among them, SNR （m，k） SNR_max represents the current signal-to-noise ratio (SNR) of the k-th sensor for the m-th physiological signal. The current SNR is the ratio of the effective signal strength to the noise strength; a higher value indicates a clearer signal. （m） The maximum signal-to-noise ratio (SNR) of the m-th physiological signal is the highest SNR that the m-th physiological signal can achieve under ideal conditions (no interference, optimal sensor performance) (as the standard for perfect quality).

[0046] Understandably, due to the initial spatial attention weights as （m） =exp(e_s （m） ) / Σ n exp(e_s （n） Therefore, it can be known that, in reality, the target space attention weight 'a' is... （m） It is based on [(1-Pfail （m，1） )·R （m，1） +(1-Pfail （m，2） )·R （m，2） ] / [((1-Pfail （n，1） )·R （n，1） +(1-Pfail （n，2） )·R （n，2） [The initial spatial attention weights as] （m） The result was obtained after correction.

[0047] In this embodiment, the time window length T used to calculate the temporal attention weight and spatial attention weight is set to 30 seconds, which can capture the short-term changing trend of physiological signals.

[0048] S203, a weighted fusion is performed based on the temporal attention weight and the target spatial attention weight to obtain a fused feature vector. In this embodiment, the weighted fusion of the temporal attention weight and the target spatial attention weight integrates the features corresponding to different time steps and different physiological signals into a final fused feature vector. Specifically, the calculation formula for the fused feature vector is as follows: F=Σ t Σ m at （t） ·as （m） ·f_t （t，m） in: F is the fusion feature vector; f_t （t，m） Let be the original feature vector of the m-th physiological signal at the t-th time step; at （t） Let be the temporal attention weight at time step t; as （m） Let be the target space attention weight for the m-th physiological signal; Σ t This refers to summing over all time steps, where t represents each time step; Σ m This refers to summing all physiological signal types, where m represents each type of physiological signal.

[0049] S30, determine the passenger state type based on the fused feature vector, and execute the state adjustment operation corresponding to the passenger state type. The state adjustment operation is a preset adjustment operation associated with each passenger state type, or a preset adjustment operation adjusted according to passenger preferences. The preset adjustment operation includes, but is not limited to, one or more of the following: adjusting the seat angle, adjusting the ambient light color temperature, releasing fragrance, and playing voice and / or music. The state adjustment operation may also include dimensional seat vibration massage for physical relaxation, temperature and humidity adjustment for environmental comfort control, and AR / VR visual guidance for expanding cognitive behavior, thus enhancing the adjustment effect and providing a more comprehensive personalized comfort improvement solution. In this embodiment, preset adjustment operations associated with each passenger state type can be preset. However, preset adjustment operations can also be adjusted according to actual needs, such as passenger preferences. That is, passengers can adjust the preset adjustment operations associated with each passenger state type within a limited adjustment range according to their actual preferences, and then store the adjustments. In this case, after determining the passenger state type, the preset adjustment operation adjusted according to passenger preferences can be directly retrieved and executed.

[0050] For example, passengers can adjust the seat angle based on their height and preferences, with a corresponding adjustment range of ±5°; passengers can adjust the ambient light color temperature based on their preferences, with a corresponding adjustment range of 2700K-6500K; passengers can adjust the type and concentration of fragrance to be released based on their historical preferences, with a corresponding adjustment range of menthol or linalool, and the concentration adjusted according to their state; personalized dialogue can be generated based on passengers' historical preferences and language habits, which constitutes a limited adjustment range, and passengers can choose one of them to play within this range; music with a match rate >90% (the match rate can be determined based on the music preference keywords selected by the passenger) can be used as a limited adjustment range based on the passenger's historical music listening records and current state, and passengers can choose to play music within this limited adjustment range. In this way, based on passenger preferences and historical data, it can be ensured that the final adjustment operations performed by the vehicle conform to passenger preferences.

[0051] In one embodiment, step S30, performing a state adjustment operation corresponding to the passenger state type, includes: When the passenger status type is emergency, an emergency adjustment operation is performed. The emergency adjustment operation includes one or more of the following operations: The seat automatically adjusts to a flat position (0° recline). Adjust the ambient lighting to a warm tone (2700K); Releases a high concentration of mint fragrance (100% concentration); Audio message: We have detected that you may be feeling unwell. Please take deep breaths. We are adjusting your environment. Play soothing music with a personalization match of >90%.

[0052] When the passenger's condition is classified as severe discomfort, a high-intensity adjustment operation is performed; the high-intensity adjustment operation includes one or more of the following operations: Slightly adjust the seat (adjust the tilt angle by 5°, tilt it back); The ambient light color temperature gradually changes to 3000K (warm tone); Releases a medium concentration of mint fragrance (60% concentration); A voice message was played: We have detected that you may be feeling unwell. We suggest you adjust your breathing rhythm. Play soothing music with a personalization match of >85%.

[0053] When the passenger's condition is classified as moderate discomfort, a moderate adjustment operation is performed; the moderate adjustment operation includes one or more of the following operations: Seat fine-tuning (tilt adjustment 3°); The ambient light color temperature gradually changes to 4000K (neutral tone); Releases a low concentration of linalool fragrance (30% concentration); The audio message reads: "We suggest you relax your body and take a deep breath." Play relaxing music with a personalization match of >80%.

[0054] When the passenger's condition is classified as mild discomfort, a mild adjustment operation is performed; the mild adjustment operation includes one or more of the following operations: Seat fine-tuning (tilt adjustment 2°); The ambient light color temperature gradually changes to 5000K (leaning towards a cool tone); Releases a low concentration of linalool fragrance (15%). Play relaxing music with a personalization match of >75%.

[0055] When the passenger status type is normal, the current environmental parameters around the passenger remain unchanged. That is, when the passenger status type is normal, the current environmental parameters are maintained, and the passenger status is not adjusted.

[0056] Understandably, in this invention, the classification of passenger condition types is not limited to the five levels mentioned above. The number of levels can be set according to needs. For example, it can be divided into 3 levels (mild, moderate, and severe) or 7 more granular levels. It is not limited here, as long as different condition adjustment thresholds are set according to the severity of the passenger condition (the condition adjustment threshold represents the adjustment intensity of the condition adjustment operation. For example, the condition adjustment threshold includes the angle threshold of seat adjustment, the color temperature threshold of ambient light adjustment, the type and concentration of released fragrance, the tone and timbre of played voice or music, etc.), to ensure that the adjustment is effective and not excessive.

[0057] In some embodiments, when the vehicle travels to different countries or regions, the fragrance type or music style in the status adjustment operation can be automatically adjusted according to the cultural differences of the different countries or regions. The vehicle's travel time can also be counted in real time, and the duration of the status adjustment operation can be adjusted according to the travel time to maintain the mapping relationship between passenger status and status adjustment operation, adapting to different operational needs.

[0058] In this embodiment, after acquiring multimodal physiological signals through sensing components, the temporal and spatial features extracted from the multimodal physiological signals are fused based on the failure probability of the sensing components to obtain a fused feature vector. During the fusion of the fused feature vector, the temporal correlation of each physiological signal in the multimodal physiological signals (i.e., the importance of the same physiological signal varies at different time points) and the spatial correlation between different physiological signals (i.e., the coupling relationship between different physiological signals) are fully considered based on the temporal features. Therefore, the passenger state type determined by the fused feature vector comprehensively considers spatiotemporal characteristics, improving the accuracy of passenger state type classification. Simultaneously, since the failure probability of the sensing components is also considered during the fusion of the fused feature vector, the robustness of passenger state type classification is also improved. Consequently, with improved accuracy and robustness in passenger state type classification, the final state adjustment operation performed based on the passenger state type will be more closely matched to the passenger's current actual state, improving the adjustment effect and passenger experience. Under laboratory conditions, for specific scenarios (such as passengers sitting still and smooth road conditions), the passenger state adjustment method of this invention can achieve an accuracy rate of over 95% in identifying passenger state types. In more complex and variable real-world environments, the accuracy rate for identifying core passenger states (such as emergency situations and severe discomfort) can also be maintained at over 90%. Therefore, the overall classification accuracy of passenger state types using this invention can reach a practical level of 85%-92%, thus ensuring system reliability.

[0059] In one embodiment, each of the multimodal physiological signals is measured by at least two sensors in the sensing component; each physiological signal corresponds to two sensors, namely a main sensor and a backup sensor.

[0060] Further, in step S30, determining the passenger state type based on the fused feature vector includes: classifying the fused feature vector by a preset learning model to obtain the passenger state type; the preset learning model is trained based on a preset training sample set and a loss function, wherein, in the training samples of the preset training sample set, some physiological signals measured by the sensing component are randomly masked according to the failure probability of the sensing component; the loss function is designed based on the failure probability of the sensing component.

[0061] During the vehicle's operation, multimodal physiological signals are collected in real time. After processing in steps S10 and S20 to obtain a fused feature vector, the fused feature vector is input into a pre-trained preset learning model. The model outputs a probability distribution corresponding to all passenger state types (e.g., five categories: emergency state, severe discomfort state, moderate discomfort state, mild discomfort state, and normal state, with the degree of discomfort decreasing progressively). The preset learning model is as follows: P(state=c|F)=Softmax(Wc T F+bc) in: F is the fusion feature vector; P(state=c|F) refers to the probability of the c-th passenger state type under the condition of fusing feature vector F (each c value corresponds to a passenger state type, such as c=1 corresponding to the probability of "normal state").

[0062] Softmax is the activation function; Wc T This is the transpose of the model parameter matrix corresponding to the c-th passenger state type (the weights learned during model training). bc The model bias term corresponding to the c-th passenger state type (parameters learned during model training).

[0063] Then, the passenger state type with the highest probability is used as the passenger state type determined and output by the preset learning model.

[0064] Understandably, during the training process of the aforementioned pre-defined learning model, it is necessary to consider the failure probability of the sensing components, perform sample masking, and design a loss function to improve the robustness of the model to sensor failures in the sensing components.

[0065] In one embodiment, before classifying the fused feature vector using a preset learning model to obtain the passenger state type, the process includes: Multiple sets of training data are acquired, each set including all types of physiological signals required for the multimodal physiological signals. The training data can be pre-stored in a database for easy retrieval. The training data in the database can be measured by a sensing component or simulated, but the simulated training data will also be associated with the sensing component. After acquiring the training data, all training data can be filtered, denoised, and normalized to remove outliers and noise. Then, all training data are divided into a training set (70%), a validation set (15%), and a test set (15%).

[0066] In each set of training data, physiological signals corresponding to at least one sensor are randomly masked based on the failure probability of the sensing components. Training samples are generated based on the masked training data, and sample state types are labeled for each training sample. A training sample set is then generated based on all training samples and their corresponding sample state types. Specifically, during the training of a preset learning model using training samples from the preset training sample set, it is necessary to randomly mask data measured by some sensors in the sensing components (including features extracted from physiological signals measured by the main sensor, etc.) according to the failure probability of the sensing components, in order to simulate the failure of some sensors. That is, since each type of physiological signal needs to be detected by both the main sensor and the backup sensor, during the random masking process, the physiological signals corresponding to the main sensor and / or the backup sensor for one or more physiological signals may be randomly masked (the masked content here includes the part of the physiological signal that needs to be masked in the training data and all its corresponding features, such as the time features, spatial features, fusion feature vectors, etc., corresponding to the part of the physiological signal that needs to be masked in the training data). Understandably, for each training sample (i.e., each set of training data), the failure probability P_fail of the main sensor for the m-th physiological signal at the current time t is calculated. （m，1） The data corresponding to the main sensor in its training data is masked, and the failure probability P_fail of the backup sensor for the m-th physiological signal at the current time t is calculated. （m，2） Data corresponding to the backup sensor is masked. Understandably, if both the primary and backup sensors are masked, the historical average value corresponding to that value is used to fill in the masked data.

[0067] Then, training samples can be generated based on the masked training data, and the sample state type can be labeled for the training samples (e.g., emergency state, severe discomfort state, moderate discomfort state, mild discomfort state, normal state). Then, a training sample set can be generated based on all training samples and their corresponding labeled sample state types.

[0068] A robustness loss term is determined based on the failure probability of the sensing component. Based on this robustness loss term, a loss function is constructed using weighted cross-entropy loss. Specifically, the loss function can be designed using weighted cross-entropy loss as follows: L=-Σc yc log( c)+α·Lrobust in: yc is the real label; for example, if the c-th passenger state type is a real state, yc=1, otherwise yc=0.

[0069] c represents the predicted probability; that is, the probability of the c-th passenger state type predicted by the pre-defined learning model.

[0070] α is the robustness weight, set to 0.1; Lrobust is the robust loss term: Lrobust=ΣmΣkPfail （m，k） ·||F （m，k） -Fpred （m，k） || 2 in: Pfail （m，k） Let be the failure probability of the k-th sensor for the m-th physiological signal; F （m，k） The physiological signal characteristics actually collected by the k-th sensor for the m-th physiological signal include the temporal and spatial features extracted from the collected physiological signals. Fpred （m，k） The physiological signal features corresponding to the k-th sensor for the m-th physiological signal predicted by the preset learning model.

[0071] The deep neural network is trained using the training sample set and the loss function to obtain a preset learning model. The Adam optimizer can be used, with an initial learning rate of 0.001 and a cosine annealing strategy; the batch size is 32, and the number of training epochs is 100; the loss function is calculated, and the model parameters are updated via backpropagation. All training data is divided into a training set (70%), a validation set (15%), and a test set (15%). After training the preset learning model using the training set, its performance can be evaluated using the validation set, and hyperparameters can be adjusted; the final model performance can be evaluated using the test set to ensure an accuracy of over 95%. In this embodiment, training is based on a deep neural network model with a 3-layer hidden layer structure, capable of learning complex feature representations while avoiding overfitting; the dropout rate is set to 0.2-0.3 to prevent overfitting; the initial learning rate is 0.001, and a cosine annealing strategy is used to ensure model convergence; the batch size is set to 32 to balance training speed and model performance.

[0072] The aforementioned pre-defined learning model is constructed and trained using a deep neural network with the following structure: Input layer: fused feature vector F, dimension 128; Hidden layer 1: fully connected layer, 256 neurons, ReLU activation function, Dropout rate 0.3; Hidden layer 2: fully connected layer, 128 neurons, ReLU activation function, Dropout rate 0.3; Hidden layer 3: fully connected layer, 64 neurons, ReLU activation function, Dropout rate 0.2; Output layer: fully connected layer, 5 neurons, corresponding to 5 passenger states, Softmax activation function.

[0073] Understandably, in some embodiments, other model architectures can also be used to train deep learning models. For example, a preset learning model can be obtained by replacing deep neural networks with support vector machines (SVM) and manual feature engineering, or by replacing deep neural networks with random forest ensemble learning or lightweight MobileNetV3 architecture. As long as the ability to classify passenger state types can be maintained, the accuracy and computational efficiency can be balanced by adjusting the model complexity.

[0074] Furthermore, in step S30, after determining the passenger state type based on the fused feature vector, the method further includes: performing differential privacy processing on the fused feature vector to obtain a target feature vector with added Laplace noise, and then storing the target feature vector in a preset storage location.

[0075] In some embodiments, a lightweight differential privacy processing algorithm (ε=0.5) is used to add Laplace noise to the fused feature vector F to achieve differential privacy processing. Afterwards, only the encrypted target feature vector can be stored in a preset storage location (e.g., the cloud) to ensure compliance with EU data protection regulations and aviation data security specifications. Specifically, the fused feature vector is differentially privacy processed using the following formula to obtain the target feature vector with added Laplace noise: Fprivate = F + Lap(ΔF / ε) in: Fprivate is the target feature vector.

[0076] F is the fused feature vector.

[0077] ΔF represents the sensitivity of the fused feature vectors. ΔF can be determined based on the range of values ​​for the feature vectors, for example, 1.0.

[0078] ε represents the privacy budget, which can be set according to needs. For example, setting ε=0.5 can strike a balance between privacy protection and data availability. A smaller privacy budget ε results in stronger privacy protection.

[0079] Lap(λ) is a Laplace distribution.

[0080] In one embodiment, such as Figure 4 As shown, after step S30, that is, after performing the state adjustment operation corresponding to the passenger state type, the following steps S40-S50 are included: S40, acquire the feedback physiological signals collected in real time by the sensing component after the state adjustment operation is performed; wherein, the feedback physiological signals collected in real time by the sensing component can be statistically analyzed, and the feedback physiological signals on a short time scale (τ_s) (e.g., when the vehicle is an aircraft, the feedback physiological signals of the aircraft's most recent 3 flights) are determined as short-term feedback data, which can be used to quickly respond to changes in passenger state. The feedback physiological signals on a long time scale (τ_l) (e.g., when the vehicle is an aircraft, the feedback physiological signals of the aircraft's most recent 7-15 flights) are determined as long-term feedback data, which can be used to establish a long-term regulatory physiological baseline.

[0081] S50, determine short-term regulatory feedback indicators and long-term regulatory physiological baselines based on the feedback physiological signals, and adjust the state regulation operation based on the short-term regulatory feedback indicators and long-term regulatory physiological baselines. That is, determine short-term regulatory feedback indicators based on the feedback physiological signals (short-term feedback data) at the short-term time scale (τ_s), and determine long-term regulatory physiological baselines based on the feedback physiological signals (long-term feedback data) at the long-term time scale (τ_l).

[0082] Specifically, the short-term adjustment feedback index Fshort(t) is calculated according to the following formula: Fshort(t)=(1 / 3)Σiαi·Si in: Fshort(t) is the short-term adjustment feedback index corresponding to the current number of flights (or trips) t; t represents the current number of flights (or trips); i represents the i-th flight (or voyage) in history. Si is the feature vector corresponding to the feedback physiological signal of the i-th flight (or journey) in history; αi is the time decay weight, and the newer the flight (or flight), the larger the value of αi; where αi=exp(-(ti) / τ_s); τ_s is the short-term time scale, which can be set to 1-5 times according to actual needs to adapt to high-frequency flight (or flight) scenarios, for example, τ_s=3.

[0083] The long-term regulated physiological baseline Blong(t) is calculated using the following formula: Blong(t)=(1 / N)Σiβi·Si in: Blong(t) is the long-term regulated physiological baseline corresponding to the current flight (or driving) number t; N represents the size of the long-term time window (e.g., 7-15 flights or trips correspond to 7-15 times). i represents the i-th flight (or voyage) in history. Si is the feature vector corresponding to the feedback physiological signal of the i-th flight (or journey) in history; βi is the long-term decay weight, where βi=exp(-(ti) / τl); τl is the long-term time scale, which can be set to 10-20 times according to actual needs to adapt to low-frequency flight (or driving) scenarios, for example, τl=15.

[0084] Subsequently, the state regulation operation and its corresponding state regulation threshold can be adjusted based on the short-term regulatory feedback index and the long-term regulatory physiological baseline.

[0085] Specifically, the state adjustment threshold corresponding to the state adjustment operation is dynamically adjusted according to the following formula: Tintervention(c)=Tbase(c)+γ1·(Fshort(t)-Blong(t))+γ2·ΔB_long in: Tbase(c) is the baseline adjustment threshold for the c-th passenger state type; the baseline adjustment threshold is the preset state adjustment threshold for each operation in the state adjustment operation; Tintervention(c) is the state adjustment threshold after adjusting the baseline adjustment threshold for the c-th passenger state type; ΔB_long is the rate of change of the long-term regulating physiological baseline; ΔB_long = B_long(t) - B_long(t-1); B_long(t) is the long-term physiological baseline for the current number of flights (or trips) t; B_long(t-1) is the long-term regulatory physiological baseline for the t-1th flight (or journey); γ1 and γ2 are both preset adjustment coefficients; where γ1=0.3 and γ2=0.2.

[0086] The state adjustment operation is dynamically adjusted according to the following formula: Ioptimized(c) = Ibase(c) + η·( E_intervention / I) in: I_base(c) is the preset adjustment operation for the c-th passenger state type; Ioptimized(c) is the state adjustment operation after adjusting the preset adjustment operation for the c-th passenger state type.

[0087] η is the learning rate, which can be set to 0.001-0.1 to accommodate different convergence speed requirements; for example, setting η=0.01 can ensure the stability of policy optimization.

[0088] E_intervention is an indicator for evaluating the effectiveness of regulation; where: Eintervention=(1 / T)Σtwt·(S_before(t)-S_after(t)) In the above formula, S_before(t) represents the characteristics of the feedback physiological signal before the current flight (or driving) number t is adjusted; S_after(t) represents the characteristics of the feedback physiological signal after the current flight (or driving) number t is adjusted; wt is the time weight, and T is the evaluation time window.

[0089] In this embodiment, a rapid response is achieved by adjusting the feedback index in the short term, and the baseline is stabilized by adjusting the physiological baseline in the long term. The combination of the two for dual timescale learning can realize the dynamic adjustment of state regulation operations and the state regulation thresholds corresponding to each operation.

[0090] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0091] This invention also provides a vehicle including the aforementioned controller. The controller may be a server, and its internal structure diagram may be as follows. Figure 5 As shown, the controller includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. When the computer-readable instructions are executed by the processor, they implement a passenger state adjustment method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0092] In one embodiment, the controller includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the passenger state adjustment method described above. This controller corresponds one-to-one with the passenger state adjustment method described in the above embodiments. Specific limitations of the controller can be found in the limitations of the passenger state adjustment method described above, and will not be repeated here. Each module in the controller can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of the processor in the controller, or stored in software in the memory of the controller, so that the processor can call and execute the operations corresponding to each module.

[0093] In the vehicle described in the above embodiments of the present invention, after acquiring multimodal physiological signals through sensing components, the temporal and spatial features extracted from the multimodal physiological signals are fused based on the failure probability of the sensing components to obtain a fused feature vector. During the fusion of the fused feature vector, the temporal correlation of each physiological signal in the multimodal physiological signals (i.e., the importance of the same physiological signal differs at different time points) and the spatial correlation between different physiological signals (i.e., the coupling relationship between different physiological signals) are fully considered based on the temporal features. Therefore, the passenger state type determined by the fused feature vector comprehensively considers spatiotemporal characteristics, improving the accuracy of passenger state type classification. Simultaneously, since the failure probability of the sensing components is also considered during the fusion of the fused feature vector, the robustness of passenger state type classification is also improved. Furthermore, with both the accuracy and robustness of passenger state type classification improved, the final state adjustment operation performed based on the passenger state type will be more closely matched to the passenger's current actual state, improving the adjustment effect and passenger experience.

[0094] In one embodiment, a computer-readable storage medium is provided that stores computer-readable instructions thereon, which, when executed by a processor, implement the steps of the passenger state adjustment method described above.

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a readable storage medium, including non-volatile readable storage media and volatile readable storage media. When executed, the computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct memory bus RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units or modules is used as an example. In practical applications, the above functions can be assigned to different functional units or modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0097] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A passenger status adjustment method, characterized in that, include: Passengers' multimodal physiological signals are collected through sensing components; Based on the failure probability of the sensing component, the temporal and spatial features extracted from the multimodal physiological signal are fused to obtain a fused feature vector. The passenger state type is determined based on the fused feature vector, and a state adjustment operation corresponding to the passenger state type is performed.

2. The passenger status adjustment method as described in claim 1, characterized in that, The process of collecting passengers' multimodal physiological signals through sensing components includes: Obtain the frequency range of signal fluctuations for each physiological signal to be collected, and determine the collection frequency of each physiological signal based on the frequency range of signal fluctuations. The physiological signals are collected according to the respective collection frequencies using the sensing components. After preprocessing all the collected physiological signals, multimodal physiological signals are obtained.

3. The passenger status adjustment method as described in claim 1, characterized in that, Based on the failure probability of the sensing component, the temporal and spatial features extracted from the multimodal physiological signal are fused to obtain a fused feature vector, including: The temporal and spatial features of the multimodal physiological signals are extracted using a preset extraction model. A temporal attention weight is determined based on all the temporal features, an initial spatial attention weight is determined based on all the spatial features, and the initial spatial attention weight is corrected based on the failure probability and reliability index of the sensing component to obtain the target spatial attention weight. The fused feature vector is obtained by weighting and fusing the temporal attention weights and the target space attention weights.

4. The passenger status adjustment method as described in claim 1, characterized in that, Each of the multimodal physiological signals is measured by two sensors in the sensing component; Determining the passenger status type based on the fused feature vector includes: The fused feature vector is classified into states by a preset learning model to obtain the passenger state type. The preset learning model is trained based on a preset training sample set and a loss function. In the training samples of the preset training sample set, some physiological signals measured by the sensing components are randomly masked according to the failure probability of the sensing components. The loss function is designed based on the failure probability of the sensing components.

5. The passenger status adjustment method as described in claim 4, characterized in that, Before classifying the fused feature vector using a preset learning model to obtain the passenger state type, the process includes: Acquire multiple sets of training data, each set of training data including all types of physiological signals required for the multimodal physiological signals; In each set of training data, based on the failure probability of the sensing component, the physiological signals corresponding to at least one sensor in the training data are randomly masked, training samples are generated based on the masked training data, and the sample state types are labeled for the corresponding training samples. Then, a training sample set is generated based on all the training samples and their corresponding sample state types. A robustness loss term is determined based on the failure probability of the sensing component, and a loss function is constructed based on the robustness loss term using weighted cross-entropy loss. The deep neural network is trained using the training sample set and the loss function to obtain a preset learning model.

6. The passenger status adjustment method as described in claim 1, characterized in that, The execution of the state adjustment operation corresponding to the passenger state type includes: When the passenger status type is an emergency, an emergency adjustment operation is performed; When the passenger's condition is classified as severe discomfort, a high-intensity adjustment operation is performed. When the passenger's condition is classified as moderate discomfort, a moderate adjustment operation is performed. When the passenger's condition is classified as mild discomfort, a mild adjustment operation is performed. When the passenger status type is normal, the current environmental parameters around the passenger remain unchanged.

7. The passenger status adjustment method as described in claim 1, characterized in that, The state adjustment operation is a preset adjustment operation associated with each of the passenger state types, or a preset adjustment operation adjusted according to passenger preferences; the preset adjustment operation includes one or more of the following: adjusting the angle of the seat the passenger is sitting in, adjusting the ambient light color temperature, releasing fragrance, playing voice and / or music.

8. The passenger status adjustment method as described in claim 1, characterized in that, After performing the state adjustment operation corresponding to the passenger state type, the following is included: The feedback physiological signals are acquired in real time through the sensing components after the state adjustment operation is performed; Based on the feedback physiological signals, short-term regulatory feedback indicators and long-term regulatory physiological baselines are determined, and the state regulation operation is adjusted based on the short-term regulatory feedback indicators and long-term regulatory physiological baselines.

9. A vehicle, characterized in that, The system includes a controller, which includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the passenger state adjustment method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by a processor, they implement the passenger state adjustment method as described in any one of claims 1 to 8.

Images