Intelligent driving passenger state real-time monitoring system based on multi-modal data fusion

Abstract

The present application relates to the technical field of behavior feature recognition, in particular to an intelligent driving occupant state real-time monitoring system based on multi-modal data fusion, which comprises a multi-modal data extraction module, which obtains seat contact surface pressure distribution numerical values, calculates and generates contact surface pressure two-dimensional center coordinates, obtains upper body point cloud data sets, and calculates and obtains occupant torso three-dimensional centroid coordinates. In the present application, the radar residual envelope line slope value, displacement frequency drop amplitude value and seat belt tension variance value are uniformly modeled, a multi-dimensional state support relationship is constructed, and the fusion evaluation of the state probability is completed through the interaction calculation between the trust interval, so that the determination of the occupant abnormal behavior has the multi-source information collaborative support capability, and the overall continuous correlation link from the contact stress, spatial posture to physiological micro-motion characteristics is formed, so that the behavior recognition result is improved in terms of accuracy, stability and anti-interference ability.

Description

Technical Field

[0001] This invention relates to the field of behavioral feature recognition technology, and in particular to a real-time monitoring system for the status of intelligent driving occupants based on multimodal data fusion. Background Technology

[0002] Behavioral feature recognition technology focuses on analyzing and determining the action patterns, physiological changes, and interactive behaviors of target objects in specific environments. It relies on the collaborative acquisition and correlation calculation of multi-source sensor signals to uniformly model time series signals, spatial structure information, and dynamic change trends, thereby realizing the recognition of individual behavioral states.

[0003] Current technologies for behavioral feature recognition often rely on single signal sources or limited-dimensional information for state determination, typically using time-series signals or simple spatial features as the primary basis. In actual operation, this makes it difficult to simultaneously characterize the complex posture changes and micro-physiological features of occupants. For example, judging posture solely from image information is easily affected by occlusion factors, and judging state solely from pressure distribution lacks spatial structural support, leading to biased state judgments. Furthermore, in vibration environments, the technology fails to effectively distinguish between chassis vibrations and subtle human movements, easily misinterpreting environmental vibrations as behavioral feature changes, resulting in frequent false alarms. For instance, road surface undulations during vehicle movement directly interfere with signal determination. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a real-time monitoring system for the status of intelligent driving occupants based on multimodal data fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a real-time monitoring system for the status of intelligent driving occupants based on multimodal data fusion includes: The multimodal data extraction module obtains the pressure distribution values ​​of the seat contact surface, calculates and generates the two-dimensional center coordinates of the contact surface pressure, obtains the upper body point cloud dataset, and calculates and obtains the three-dimensional centroid coordinates of the occupant's torso. The occupant posture fusion module sets the three-dimensional centroid coordinates of the occupant's torso as a state variable parameter, calls the upper body tilt angle value and ischial tuberosity pressure transfer parameter to construct a spinal kinematic constraint algebraic expression, calculates and generates the posture residual covariance matrix, sets the two-dimensional center coordinates of the contact surface pressure as an observation variable parameter, and uses the observation variable parameter to calculate the error covariance gain weight addition of the posture residual covariance matrix to obtain the occupant's departure posture angle parameter; The vibration feature extraction module receives the echo continuous wave dataset, performs a two-dimensional frequency domain matrix transformation operation on the echo continuous wave dataset to obtain the distance Doppler two-dimensional frequency domain matrix, generates the chassis common mode vibration cancellation parameter, and extracts the residual signal dataset of the chassis common mode vibration cancellation parameter associated with the chest cavity boundary position through the occupant departure attitude angle parameter to obtain the occupant chest cavity displacement frequency feature parameter. The abnormal state assessment module calculates and generates heterogeneous probability basic allocation parameters based on the occupant's chest cavity displacement frequency characteristic parameters, performs calculations on the confidence intervals of each item of the heterogeneous probability basic allocation parameters, and obtains the occupant disability probability assessment results.

[0006] Preferably, the step of obtaining the three-dimensional centroid coordinates of the occupant's torso is as follows: The pressure distribution values ​​of each sampling unit's contact surface output by the seat pressure sensor array are called. The pressure coordinate correspondence is established according to the lateral and longitudinal positions of each sampling unit in the seat plane. The pressure distribution values ​​of each sampling unit's contact surface are substituted into the origin offset calculation of each order of spatial geometric moment coordinate system. The lateral offset and longitudinal offset are counted. The two-dimensional center position offset parameter of the occupant's contact surface is calculated, and the two-dimensional center coordinate of the contact surface pressure is generated. Read all point coordinates in the upper body point cloud dataset, read the head node coordinates and torso skeleton node coordinates in the occupant biomechanical topology wireframe coordinate set, calculate the nearest neighbor distance for each point coordinate, correct the spatial correspondence between point coordinates and node coordinates according to the minimum correspondence of the nearest neighbor distance, extract the registration iteration association error value, and perform three-dimensional spatial translation vector transformation in combination with the two-dimensional center coordinates of the contact surface pressure to form the position values ​​of the occupant head node and the three-dimensional mesh node of the torso. Based on the position values ​​of the occupant's head node and torso 3D mesh node, the occupant's head node position value is removed, and the position values ​​of the chest node, abdomen node, shoulder node, and spine node in the torso 3D mesh node position value are retained. The three-axis coordinate reference of each torso 3D mesh node position value is unified, and the position components of each torso 3D mesh node position value on the three coordinate axes are accumulated. The average value of the position components on the three coordinate axes is calculated to obtain the occupant's torso 3D centroid coordinates.

[0007] Preferably, the steps for obtaining the attitude residual covariance matrix are as follows: The three-dimensional centroid coordinates of the occupant's torso are set as state variable parameters. The coordinate components of the three-dimensional centroid coordinates of the occupant's torso in the anterior-posterior, left-right, and vertical directions are extracted. The forward flexion angle, lateral tilt angle, and backward tilt angle components in the upper body tilt angle values ​​are read. The left transfer component, right transfer component, and forward transfer component in the ischial tuberosity pressure transfer parameters are read. The spinal kinematic constraint algebraic formula is written according to the displacement relationship of the spinal segment corresponding to each coordinate component. The spinal kinematic constraint algebraic formula is substituted into the state transition equation term by term. The coordinate changes of the state variable parameters at continuous time are recursively expanded. The component differences between the current state variable parameters and the recursive results are compared. The offset and offset direction of each component difference are statistically analyzed to obtain the prior estimation error of the state variable parameters. The prior estimation error of the state variable parameters is decomposed into error components in the front-back, left-right, and vertical directions. The fluctuation amplitude, number of changes, and offset concentration interval of each error component at continuous sampling time are statistically analyzed. The component correlation relationship is established according to the corresponding position of the error components in the same direction. The synchronous offset relationship between error components in different directions is calculated. The dispersion of each error component in each direction is written into the diagonal position, and the synchronous offset relationship of error components in different directions is written into the non-diagonal position to form the attitude residual covariance matrix.

[0008] Preferably, the steps for obtaining the occupant disengagement attitude angle parameters are as follows: The two-dimensional center coordinates of the contact surface pressure are set as observation variable parameters. The lateral center coordinate components and longitudinal center coordinate components in the two-dimensional center coordinates of the contact surface pressure are extracted. The corresponding offset relationships between the lateral center coordinate components, longitudinal center coordinate components and the error components in each direction of the attitude residual covariance matrix are compared. The weight of the corresponding matrix component is increased according to the position of consistent offset, and the weight of the corresponding matrix component is compressed according to the position of opposite offset. The error covariance gain weight addition is calculated. The attitude components in each direction at the current moment are corrected according to the error covariance gain weight, and the occupant spatial attitude angle at the current moment is converted to obtain the occupant departure attitude angle parameters.

[0009] Preferably, the step of obtaining the chassis common-mode vibration cancellation parameters is as follows: The echo continuous wave dataset is received by the vehicle-mounted millimeter-wave radar. The echo amplitude of each frame is sorted according to the range sampling order and velocity sampling order of the echo continuous wave dataset. The echo change corresponding to each range unit and the echo change corresponding to each velocity unit are extracted in sequence. The echo change of each frame is mapped to the intersection of the range dimension and the velocity dimension. The frequency domain distribution values ​​corresponding to the range position and the velocity position are written frame by frame to form a matrix arrangement result with a one-to-one correspondence between the range position and the velocity position, and the range-Doppler two-dimensional frequency domain matrix is ​​obtained. The longitudinal, lateral, and vertical acceleration signal values ​​are collected from the triaxial acceleration signal values ​​by the chassis inertial measurement unit. The longitudinal, lateral, and vertical acceleration signal values ​​are mapped to the same frame position in the range-Doppler two-dimensional frequency domain matrix. The vibration disturbance direction is calculated for each range unit and the vibration disturbance amplitude is calculated for each velocity unit. The radar echo common-mode vibration component values ​​in each matrix position are canceled frame by frame. The chest cavity reflection change component not covered by the longitudinal, lateral, and vertical acceleration signal values ​​is retained to generate chassis common-mode vibration cancellation parameters.

[0010] Preferably, the steps for obtaining the occupant's thoracic displacement frequency characteristic parameters are as follows: The flexion, roll, and rotation angles from the occupant displacement posture angle parameters are called. The projection positions of the thoracic cavity boundary in the distance and angle directions are corrected according to the flexion, roll, and rotation angles. The residual change values ​​of the corresponding positions of the thoracic cavity boundary are extracted frame by frame from the chassis common-mode vibration cancellation parameters to form a thoracic cavity boundary position residual signal sequence. The periodic displacement fluctuations in the thoracic cavity boundary position residual signal sequence are statistically analyzed in chronological order. The frequency positions with the most concentrated amplitude changes and the frequency positions with the most continuous frequency changes in each continuous time period are extracted to obtain the occupant thoracic cavity displacement frequency characteristic parameters.

[0011] Preferably, the steps for obtaining the basic allocation parameter of the heterogeneous probability are as follows: Based on the occupant's chest cavity displacement frequency characteristic parameters, the frequency position values ​​and spectral amplitude values ​​corresponding to continuous sampling times are extracted. The spectral amplitude values ​​are connected in chronological order to form a radar residual envelope. The envelope change and time interval of adjacent sampling times are statistically analyzed segment by segment. The slope change value corresponding to the direction of envelope change in each segment is calculated and summarized to form the slope value of the radar residual envelope. At the same time, the frequency position values ​​of adjacent sampling times are compared, the change difference of the frequency position value in the continuously decreasing segment is statistically analyzed and accumulated to form the displacement frequency decrease amplitude value. Then, the continuous seat belt tension value output by the seat belt Hall sensor is called, and the dispersion of the continuous seat belt tension value relative to the average tension value is calculated to form the seat belt tension variance value, thus obtaining a heterogeneous parameter set. Based on the heterogeneous parameter set, the values ​​of radar residual envelope slope, displacement frequency decrease amplitude, and seat belt tension variance are read respectively. According to the position of each value in the corresponding confidence interval, the support intervals for normal state, fluctuation state, and abnormal state are divided item by item. The interval proportion and boundary offset of each value falling into each confidence interval are calculated. The interval proportion and boundary offset are written into the support share of the corresponding state, the conflicting support shares are compressed, and the total benchmark of the support shares of each state is unified to form the basic allocation parameter of heterogeneous probability.

[0012] Preferably, the steps for obtaining the occupant disability probability assessment results are as follows: The trust intervals corresponding to the support shares for normal state, fluctuation state, and abnormal state in the heterogeneous probability basic allocation parameters are read item by item. The intersection range and conflict range of each trust interval are compared in turn. The support shares corresponding to the intersection range are superimposed, and the support shares corresponding to the conflict range are deducted. The deducted support shares are redistributed to the reserved state to form the fused abnormal state support shares. The occupant abnormal behavior sudden state assessment probability value is calculated. Then, the occupant abnormal behavior sudden state assessment probability value is compared with the preset disability probability judgment threshold item by item to determine whether the occupant abnormal behavior sudden state assessment probability value reaches the preset disability probability judgment threshold, and the occupant disability probability assessment result is obtained.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, spatial distribution mapping of the pressure distribution values ​​on the seat contact surface and extraction of the two-dimensional center coordinates of the contact surface pressure allow for a quantitative representation of the force concentration location of the occupant within the seat. This is further achieved by aligning the upper body point cloud data with the biomechanical topology to derive the three-dimensional centroid coordinates of the occupant's torso, realizing a cross-dimensional correlation between contact surface force and spatial structural position. Based on this, a constraint relationship is established by introducing upper body tilt angle values ​​and ischial tuberosity pressure transfer parameters to continuously evolve and calculate state variable parameters. An attitude residual covariance matrix is ​​constructed through the correlation between error components. The two-dimensional center coordinates of the contact surface pressure are then used to adjust the weights of the error distribution, enabling dynamic correction in the spatial attitude angle calculation process. Simultaneously, frequency domain mapping of echo continuous wave data is incorporated. As a result, the triaxial acceleration signal values ​​of the chassis inertial measurement unit were introduced to cancel the common-mode vibration components, separating the small displacement signals of the chest cavity boundary region from the overall vibration background. The frequency characteristic parameters of the occupant's chest cavity displacement were extracted by the frequency change trend, thus characterizing the micro-dynamics related to breathing. Furthermore, the slope values ​​of the radar residual envelope, the amplitude values ​​of the displacement frequency decrease, and the variance values ​​of the seat belt tension were uniformly modeled to construct a multi-dimensional state support relationship. The state probability fusion evaluation was completed through interactive calculation between trust intervals, enabling the judgment of abnormal occupant behavior to have the ability to support multi-source information collaboration. The whole formed a continuous correlation link from contact force, spatial attitude to physiological micro-motion characteristics, which improved the accuracy, stability and anti-interference ability of the behavior recognition results. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the principle of the present invention; Figure 2 Schematic diagram of the principle of two-dimensional center coordinate positioning of contact surface pressure; Figure 3 The curves show the variation of heterogeneous parameters between occupant chest displacement and seatbelt tension. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Please see Figure 1-3 The present invention provides a technical solution: a real-time monitoring system for the status of intelligent driving occupants based on multimodal data fusion, comprising: The multimodal data extraction module obtains the pressure distribution values ​​of the seat contact surface, calculates and generates the two-dimensional center coordinates of the contact surface pressure, obtains the upper body point cloud dataset, and calculates and obtains the three-dimensional centroid coordinates of the occupant's torso. The occupant posture fusion module sets the three-dimensional centroid coordinates of the occupant's torso as a state variable parameter, calls the upper body tilt angle value and ischial tuberosity pressure transfer parameters to construct a spinal kinematic constraint algebraic expression, calculates and generates the posture residual covariance matrix, sets the two-dimensional center coordinates of the contact surface pressure as an observation variable parameter, and uses the observation variable parameter to calculate the error covariance gain weight addition of the posture residual covariance matrix to obtain the occupant's departure posture angle parameters; The vibration feature extraction module receives the echo continuous wave dataset, performs a two-dimensional frequency domain matrix transformation operation on the echo continuous wave dataset to obtain the distance Doppler two-dimensional frequency domain matrix, generates the chassis common mode vibration cancellation parameter, and extracts the residual signal dataset of the chest cavity boundary position associated with the chassis common mode vibration cancellation parameter through the occupant departure attitude angle parameter to obtain the occupant chest cavity displacement frequency feature parameter. The abnormal state assessment module calculates and generates basic heterogeneous probability allocation parameters based on the occupant's chest cavity displacement frequency characteristic parameters, performs calculations on the confidence intervals of each item in the basic heterogeneous probability allocation parameters, and obtains the occupant disability probability assessment results.

[0017] The steps to obtain the three-dimensional centroid coordinates of the occupant's torso are as follows: The pressure distribution values ​​of each sampling unit's contact surface output by the seat pressure sensor array are called. The pressure coordinate correspondence is established according to the lateral and longitudinal positions of each sampling unit in the seat plane. The pressure distribution values ​​of each sampling unit's contact surface are substituted into the origin offset calculation of each order of spatial geometric moment coordinate system. The lateral offset and longitudinal offset are counted. The two-dimensional center position offset parameter of the occupant's contact surface is calculated, and the two-dimensional center coordinate of the contact surface pressure is generated. Read all point coordinates in the upper body point cloud dataset, read the head node coordinates and torso skeleton node coordinates in the occupant biomechanical topology wireframe coordinate set, calculate the nearest neighbor distance for each point coordinate, correct the spatial correspondence between point coordinates and node coordinates according to the minimum correspondence of the nearest neighbor distance, extract the registration iteration association error value, and perform three-dimensional spatial translation vector transformation in combination with the two-dimensional center coordinates of the contact surface pressure to form the position values ​​of the occupant head node and the three-dimensional mesh node of the torso. Based on the position values ​​of the occupant's head node and the 3D mesh node of the torso, the position values ​​of the occupant's head node are removed, and the position values ​​of the chest node, abdomen node, shoulder node, and spine node in the 3D mesh node of the torso are retained. The three-axis coordinate reference of the position values ​​of the occupant's 3D mesh node are unified, and the position components of the position values ​​of the occupant's 3D mesh node on the three coordinate axes are accumulated. The average value of the position components on the three coordinate axes is calculated to obtain the occupant's torso 3D centroid coordinates.

[0018] Specifically, by calling the pressure distribution values ​​of the contact surfaces of each sampling unit output by the seat pressure sensor array, the seat plane is first meshed, for example, divided into a 48x48 matrix, where each unit corresponds to a pressure sensor, and each sensor unit (i, j) is assigned its physical coordinates in the seat coordinate system. The coordinate system has its origin (0, 0) at the center of the rear of the seat, with the y-axis horizontally and the x-axis vertically. The pressure distribution values ​​of the contact surfaces of each sampling unit are then used. With corresponding coordinates The data are correlated to form a pressure-coordinate mapping set. Then, the pressure distribution values ​​of the contact surface of each sampling unit are substituted into the calculation of the spatial geometric moments. Specifically, the two-dimensional center coordinates of the contact surface pressure are determined by performing a centroid calculation based on a pressure-weighted average. The calculation process is as follows: ; ; in, This represents the pressure value measured by the sensor in the i-th row and j-th column. Represents the vertical coordinate of the i-th row of sensors. Let M represent the horizontal coordinate of the j-th column sensor, and M and N be the row and column numbers of the sensor array, respectively. This is obtained through calculation. This refers to the precise position of the pressure center of mass on the seat plane. Subsequently, to quantify the degree of occupant posture deviation, a two-dimensional center position offset parameter of the occupant contact surface is calculated. This parameter is obtained by calculating the two-dimensional center coordinates of the contact surface pressure. With a preset seat reference center coordinate The comparison revealed that the reference center coordinates are pre-calibrated based on the seat's design dimensions. For example, for a seat with a width of 500mm and a depth of 450mm, its reference center coordinates can be set to (225, 250), and the offset parameter is... Finally, the calculated pressure centroid coordinates are... Output as the two-dimensional center coordinates of the contact surface pressure.

[0019] The coordinates of all points in the upper body point cloud dataset are read, along with a standard occupant biomechanical topology wireframe coordinate set containing the 3D coordinates of key nodes such as the head, neck, shoulders, and spine. An iterative nearest-point algorithm is used to register the real-time point cloud dataset with this biomechanical topology wireframe model. The registration process begins by registering each node in the topology wireframe model... Find the point with the closest Euclidean distance in a point cloud dataset. As corresponding points, an optimal 3D rigid body transformation (including rotation matrix R and translation vector t) is then solved by minimizing the sum of squared distances between all corresponding point pairs. The minimization objective function is: Where K is the total number of nodes in the topological wireframe model, this process iterates repeatedly. In each iteration, the rotation matrix R and translation vector t are updated, and the transformation is applied to the topological wireframe model until the error change between two iterations is less than a preset convergence threshold (e.g., If the maximum number of iterations is reached (e.g., 100), after the iterations converge, the final root mean square error is extracted as the registration iteration correlation error value to evaluate the registration quality. Then, the two-dimensional center coordinates of the contact surface pressure generated in the previous step are used. The registered model is then globally localized. Specifically, a three-dimensional spatial translation vector is constructed, using the two-dimensional center coordinates of the contact surface pressure as the reference position of the model on the X-axis (front-to-back direction) and Y-axis (left-to-right direction) in the vehicle coordinate system, combined with a preset seat surface height value. As the reference for the Z-axis (vertical direction), this translation vector is used to transform the entire registered topological wireframe model from its local coordinate system to the vehicle's global coordinate system, forming the position values ​​of the occupant head node and the torso 3D mesh node.

[0020] Based on the positional values ​​of the occupant's head node and torso 3D mesh nodes, a node filtering operation is first performed. By traversing all nodes and checking their biomechanical labels, all occupant head node positional values ​​labeled "head" or "neck" are removed. Simultaneously, all nodes whose labels are predefined as torso components are retained. These predefined sets of node labels include, for example, "left shoulder node," "right shoulder node," "suprasternal notch node," "xiphoid process node," "7th cervical vertebra node," "8th thoracic vertebra node," and "3rd lumbar vertebra node," forming a single node. A subset of torso-related nodes is included. Next, it is confirmed that the position values ​​of all retained torso 3D mesh nodes are under a unified vehicle global three-axis coordinate reference, requiring no additional coordinate transformation. Then, the geometric centers of these torso nodes are calculated as approximate estimates of the occupant torso's 3D centroid coordinates. The calculation process is as follows: the position components of all retained torso 3D mesh node position values ​​on the X, Y, and Z coordinate axes are summed, and then the sum for each axis is divided by the total number of retained torso nodes. Specifically, if the set of retained torso nodes is... ,in If the number of trunk nodes is given, then first calculate the sum of the components of the three coordinate axes. , , Then calculate the average value of each component, i.e. , , Finally, these three average values ​​are combined into a three-dimensional coordinate point. The three-dimensional centroid coordinates of the occupant's torso are obtained.

[0021] The steps to obtain the attitude residual covariance matrix are as follows: The three-dimensional centroid coordinates of the occupant's torso are set as state variable parameters. The coordinate components of the three-dimensional centroid coordinates of the occupant's torso in the anterior-posterior, lateral, and vertical directions are extracted. The forward flexion, lateral tilt, and backward tilt components in the upper body tilt angle values ​​are read. The left-side transfer, right-side transfer, and forward transfer components in the ischial tuberosity pressure transfer parameters are read. The spinal kinematic constraint algebraic formula is written according to the displacement relationship of the spinal segment corresponding to each coordinate component. The spinal kinematic constraint algebraic formula is substituted into the state transition equation term by term. The coordinate changes of the state variable parameters at continuous time are recursively expanded. The component differences between the current state variable parameters and the recursive results are compared. The offset and offset direction of each component difference are statistically analyzed to obtain the prior estimation error of the state variable parameters. The prior estimation error of the state variable parameters is decomposed into error components in the front-back, left-right, and vertical directions. The fluctuation amplitude, number of changes, and offset concentration interval of each error component at continuous sampling time are statistically analyzed. The component correlation relationship is established according to the corresponding position of the error component in the same direction. The synchronous offset relationship between error components in different directions is calculated. The dispersion of each error component in each direction is written into the diagonal position, and the synchronous offset relationship of error components in different directions is written into the non-diagonal position to form the attitude residual covariance matrix.

[0022] Specifically, the three-dimensional centroid coordinates of the occupant's torso are set as state variable parameters. ,in The coordinate components of the occupant's three-dimensional torso center of mass in the vehicle coordinate system in the front-back, left-right, and vertical directions are determined, and these state variable parameters are extracted. Simultaneously, the upper body tilt angle values, including the forward flexion angle component, are read from the onboard camera or inertial measurement unit (IMU). yaw angle components and pitch component And ischial tuberosity pressure transfer parameters obtained from the analysis of the seat pressure sensor array, including the left-side transfer component. Right-side transfer component and forward transfer components Based on these inputs, a spinal kinematic constraint algebraic expression describing the relationship between posture change and center of mass displacement is constructed, and this constraint relationship is integrated into a linear state transition equation to transition from the previous time step. State derivation for the current moment The prior state estimate is given by the equation: ,in, These are the state variable parameters after observation and correction from the previous moment. It is the state transition matrix, usually set as a 3x3 identity matrix, indicating that the position of the center of mass remains unchanged in the absence of external forces or active motion. It is the control input vector, composed of the acquired tilt angle value and pressure transfer parameters, i.e. ,and This is the control matrix, whose internal elements are transformation coefficients obtained from statistical analysis of a large amount of driver posture data. These coefficients are used to map changes in angle and pressure into displacements in the centroid coordinates. For example, the first row of matrix B might be represented as... ,in and The calibration coefficients are used to calculate the prior state estimate at the current time using this state transition equation. Then, it is compared with the three-dimensional centroid coordinates of the occupant's torso obtained by actual measurement and calculation at the current moment. Perform a component-by-component comparison and calculate the difference vector between the two. The prior estimation error of the state variable parameters is obtained.

[0023] Decompose the prior estimation error vector of the state variable parameters obtained in the previous process There are three independent error components, namely the forward and backward direction error components. Left and right directional error components and vertical error components Next, statistical analysis is performed on the three error component sequences within a continuous time window (e.g., collecting the most recent 200 sampling points at a sampling frequency of 20Hz). First, the fluctuation amplitude of each error component sequence is calculated, specifically by calculating the standard deviation of each sequence within the time window. Then, the number of times the sign of each component sequence changes within the window is counted as the number of changes. Using histogram analysis, the offset range from -5cm to +5cm is divided into 50 intervals, and the interval where each component value falls with the highest frequency is determined as the offset concentration interval. Subsequently, the correlation between error components is calculated based on these statistical characteristics. The variance of each error component sequence is calculated to quantify its dispersion, and the covariance between each pair of error component sequences in different directions is calculated to quantify their synchronous offset relationship. For example... and The covariance between them is calculated by We obtain, where N is the number of samples within the window, i.e., 200. and The variances of the two components within the window are respectively the average values. A positive covariance value indicates that the errors in the two directions tend to change in the same direction. Finally, the variances of the error components in each direction are written into the diagonal positions of a 3x3 matrix, and the covariances between error components in different directions are written into the corresponding off-diagonal positions, forming the attitude residual covariance matrix. Its specific form is as follows: ; in Represents variance. Represents covariance.

[0024] The steps for obtaining the occupant disengagement attitude angle parameters are as follows: The two-dimensional center coordinates of the contact surface pressure are set as the observed variable parameters. The lateral and longitudinal center coordinate components of the two-dimensional center coordinates of the contact surface pressure are extracted. The corresponding offset relationships between the lateral and longitudinal center coordinate components and the error components in each direction of the attitude residual covariance matrix are compared. The weight of the corresponding matrix component is increased according to the position of consistent offset, and the weight of the corresponding matrix component is compressed according to the position of opposite offset. The error covariance gain weight addition is calculated. The attitude components in each direction at the current moment are corrected according to the error covariance gain weight, and the occupant spatial attitude angle at the current moment is converted to obtain the occupant departure attitude angle parameters.

[0025] Specifically, the two-dimensional center coordinates of the contact surface pressure Set as observation variable parameter And extract its longitudinal center coordinate components. and horizontal center coordinate components Establish an observation model that projects three-dimensional state variable parameters onto a two-dimensional observation space. ,in The true three-dimensional centroid coordinates to be estimated, and the observation matrix. Defined as , It is observation noise, and its covariance matrix is... The calibration accuracy of the pressure sensor array is preset. For example, if the overall standard deviation of the sensor positioning error is 3mm in both directions, then... It is a diagonal matrix with diagonal elements of 1. Next, the error covariance gain weighted summation calculation is performed, that is, the Kalman gain is calculated. Its formula is ,in It is the attitude residual covariance matrix calculated in the previous process. It is the transpose of H. This represents the matrix inversion, and the gain. Based on the uncertainty predicted by the model (by (embodied) and measurement uncertainty (by) (This reflects) the dynamic allocation of weights, followed by the correction of the prior state estimate based on the error covariance gain weights. The calculation formula is as follows: The corrected posterior state estimate is obtained, which is the three-dimensional centroid coordinates of the occupant's torso at the current moment. Finally, the corrected centroid coordinates are converted into the occupant spatial attitude angles at the current moment through inverse kinematics, such as the flexion attitude angles. Calculated as Lateral attitude angle Calculated as ,in, These are the components of the corrected three-dimensional centroid coordinates of the occupant's torso. It is a pre-calibrated reference value for the three-dimensional center of mass coordinates of the occupant's torso in a standard neutral sitting posture. This reference value is obtained by collecting center of mass data from multiple test subjects of different body sizes in a standard driving sitting posture and calculating their average value. For example, it can be set as (250mm, 0mm, 600mm), with rotational attitude angle. The occupant dismount attitude angle parameters are obtained by analyzing the rate of change of the difference in the lateral coordinates of the center of mass and the center of pressure at continuous time intervals and combining these angles.

[0026] The steps for obtaining the common-mode vibration cancellation parameters of the chassis are as follows: The echo continuous wave dataset is received by the vehicle-mounted millimeter-wave radar. The echo amplitude of each frame is sorted according to the range sampling order and velocity sampling order of the echo continuous wave dataset. The echo change corresponding to each range unit and the echo change corresponding to each velocity unit are extracted in sequence. The echo change of each frame is mapped to the intersection of the range dimension and the velocity dimension. The frequency domain distribution values ​​corresponding to the range position and the velocity position are written frame by frame to form a matrix arrangement result with a one-to-one correspondence between the range position and the velocity position, and the range-Doppler two-dimensional frequency domain matrix is ​​obtained. The longitudinal, lateral, and vertical acceleration signal values ​​are collected by the chassis inertial measurement unit. These values ​​are then mapped to the same frame position in the range-Doppler two-dimensional frequency domain matrix. The vibration disturbance direction is calculated for each range unit, and the vibration disturbance amplitude is calculated for each velocity unit. The radar echo common-mode vibration component values ​​in each matrix position are canceled frame by frame. The thoracic reflection change component not covered by the longitudinal, lateral, and vertical acceleration signal values ​​is retained to generate chassis common-mode vibration cancellation parameters.

[0027] Specifically, the vehicle-mounted millimeter-wave radar receives a continuous echo dataset, which is a collection of the radar's raw IQ data. First, each frame of IQ data is processed by performing a two-dimensional fast Fourier transform (2D-FFT). The FFT is performed on the sampling points in the fast time dimension (range dimension) to obtain the target's range information, and then on the sampling points in the slow time dimension (Doppler dimension) to obtain the target's velocity information. This process converts the time-domain signal into a frequency-domain signal, generating a complex matrix. Then, the magnitude of each element in this complex matrix is ​​calculated to obtain the echo energy (i.e., amplitude) of each range-velocity unit. This maps the original echo variation to the range and velocity dimensions, generating a dataset of size [size missing] frame by frame. A real matrix, where This is the number of distance units (e.g., 256). This is the number of Doppler velocity units (e.g., 128), and each element in the matrix... Representing the signal strength in the i-th distance unit and the j-th velocity unit, this matrix is ​​the two-dimensional frequency domain matrix of the range-Doppler.

[0028] The chassis inertial measurement unit (IMU) acquires triaxial acceleration signal values ​​at a frequency synchronized with the millimeter-wave radar frame rate (e.g., 20Hz), and extracts the longitudinal acceleration signal values ​​from these values. lateral acceleration signal value Vertical acceleration signal value These acceleration signal values ​​are mapped to the range-Doppler two-dimensional frequency domain matrix frame closest to their timestamp. Then, common-mode vibration analysis is performed on each pixel (i.e., each range-velocity unit) in each frame of the range-Doppler two-dimensional frequency domain matrix. First, the triaxial acceleration of the chassis is projected onto the radar's line-of-sight direction through coordinate transformation. The expected Doppler frequency shift caused by vehicle vibration in each range unit is calculated. This frequency shift is the common-mode vibration component. For example, for the i-th range unit, the velocity component caused by its vibration... According to Integrating and considering the radar installation angle, the corresponding Doppler frequency shift is estimated to be... ,in It is the radar wavelength. Then, in the i-th row (corresponding to the i-th range cell) of the range-Doppler two-dimensional frequency domain matrix, the energy is translated along the Doppler axis by the translation amount. The corresponding number of velocity units means that this operation is equivalent to canceling the false velocities of all static targets (relative to the vehicle) caused by chassis vibration. For dynamic targets, their velocities will also be corrected accordingly. After processing, the energy concentration band caused by chassis common-mode vibration in the matrix will be pulled to near the zero Doppler axis, while the small Doppler frequency shift caused by the occupant's chest micro-movements (such as breathing and heartbeat) will be retained as residual signals. These chest reflection change components not covered by the acceleration signal constitute the chassis common-mode vibration cancellation parameters.

[0029] The steps for obtaining the occupant's thoracic cavity displacement frequency characteristic parameters are as follows: The flexion, roll, and rotation angles from the occupant disengagement posture angle parameters are called. The projection positions of the thoracic cavity boundary in the distance and angle directions are corrected according to the flexion, roll, and rotation angles. The residual change values ​​of the corresponding positions of the thoracic cavity boundary are extracted frame by frame from the chassis common mode vibration cancellation parameters to form a thoracic cavity boundary position residual signal sequence. The periodic displacement fluctuations in the thoracic cavity boundary position residual signal sequence are statistically analyzed in chronological order. The frequency positions with the most concentrated amplitude changes and the frequency positions with the most continuous frequency changes in each continuous time period are extracted to obtain the occupant thoracic cavity displacement frequency characteristic parameters.

[0030] Specifically, it calls the occupant dismount posture angle parameters generated in the previous steps, including the flexion posture angle. Lateral attitude angle and rotational attitude angle Based on these angular parameters and combined with a pre-defined geometric model of the relative position of the occupant and the radar, the three-dimensional spatial boundary of the occupant's chest cavity in the radar coordinate system under the current attitude is accurately calculated. This model approximates the chest cavity as an ellipsoid, with its center position and orientation changing with the attitude angle. Through coordinate rotation transformation, the chest cavity boundary is projected onto the radar's range-angle two-dimensional plane, determining a region of interest (ROI). This ROI covers all range and angle cells on the chest cavity surface that may generate radar echoes. Then, from the chassis common-mode vibration cancellation parameters (i.e., the range-Doppler two-dimensional frequency domain matrix sequence after vibration compensation), frame by frame, all residual signal amplitudes within this dynamic ROI are extracted and integrated to form a single time series, namely the chest cavity boundary position. A residual signal sequence is generated, reflecting the minute displacement of the chest cavity surface relative to the vehicle over time. Subsequently, a short-time Fourier transform (STFT) is applied to this time series for time-frequency analysis. A sliding window is set (e.g., window length of 5 seconds, overlap rate of 50%), and the spectrum within each window is calculated to obtain a time-frequency spectrum. Finally, the spectral characteristics of each time period (each window) are analyzed on this time-frequency spectrum to find the frequency positions with the most concentrated amplitude, which usually correspond to the main frequency of breathing or heartbeat. At the same time, by tracking the change trajectory of the main frequency peak within a continuous time window, the frequency positions with the most continuous frequency changes are extracted. These frequency values ​​and their corresponding spectral density amplitudes are output as the frequency characteristic parameters of the occupant's chest cavity displacement.

[0031] The steps to obtain the basic assignment parameter of heterogeneous probability are as follows: Based on the frequency characteristic parameters of the occupant's chest cavity displacement, the frequency position values ​​and spectral amplitude values ​​corresponding to the continuous sampling time are extracted. The spectral amplitude values ​​are connected in chronological order to form a radar residual envelope. The envelope change and time interval of adjacent sampling time are statistically analyzed segment by segment. The slope change value corresponding to the direction of envelope change of each segment is calculated and summarized to form the slope value of the radar residual envelope. At the same time, the frequency position values ​​of adjacent sampling time are compared, the change difference of the frequency position value in the continuously decreasing segment is statistically analyzed and accumulated to form the displacement frequency decrease amplitude value. Then, the continuous seat belt tension value output by the seat belt Hall sensor is called, and the dispersion of the continuous seat belt tension value relative to the average tension value is calculated to form the seat belt tension variance value, thus obtaining a heterogeneous parameter set. Based on the heterogeneous parameter set, the values ​​of radar residual envelope slope, displacement frequency decrease amplitude, and seat belt tension variance are read respectively. According to the position of each value in the corresponding confidence interval, the support intervals for normal state, fluctuation state, and abnormal state are divided one by one. The interval proportion and boundary offset of each value falling into each confidence interval are calculated. The interval proportion and boundary offset are written into the support share of the corresponding state, the conflicting support shares are compressed, and the total benchmark of the support shares of each state is unified to form the basic allocation parameter of heterogeneous probability.

[0032] Specifically, based on the occupant's thoracic cavity displacement frequency characteristic parameters, continuous sampling moments obtained from time-frequency analysis are extracted within a time window of length T (e.g., 5 seconds). Corresponding frequency peak position value and spectral amplitude values All spectral amplitude values Connecting them in chronological order forms the radar residual envelope. Then, by calculating the sampling times of two adjacent sampling moments... The slope of the envelope between A slope sequence is obtained, and the standard deviation of this slope sequence within the time window T is calculated. This standard deviation is used as the slope value of the radar residual envelope. A larger standard deviation indicates drastic and irregular changes in respiratory or heart rhythm. Simultaneously, the numerical sequence of frequency peak positions is examined. The algorithm searches for a segment where the frequency continuously decreases over N consecutive sampling points (e.g., N=5). If such a segment is found, the total frequency decrease within that segment is calculated. This accumulated value is used as the magnitude of the displacement frequency decrease, which reflects the degree of rapid slowing of breathing or heart rate. Next, the continuous seatbelt tension value output by the Hall sensor installed in the seatbelt retractor or latch is invoked. Within the same time window T, first calculate the average value of the tensile force during this period. Then, the average of the sum of squares of the differences between the tension value and the average value at each sampling point is calculated, that is, the variance of the tension sequence is calculated, forming the seat belt tension variance value. This value reflects the degree of impact or struggle of the occupant's body on the seat belt. Finally, the three indicators, namely the calculated radar residual envelope slope value, displacement frequency drop amplitude value, and seat belt tension variance value, are combined to obtain a heterogeneous parameter set.

[0033] Based on the heterogeneous parameter set, the Dempster-Shafer Theory rule is applied to assign support for each parameter to different crew states (normal, fluctuating, abnormal). First, for each parameter (radar residual envelope slope value)... , Displacement frequency decrease amplitude value Seat belt tension variance The system defines three confidence intervals for different states. The thresholds for these intervals are obtained through statistical analysis of a large amount of data collected from simulated and real-world driving scenarios. For example, for the radar residual envelope slope value, its normal state support interval might be [0, 0.5], the fluctuating state support interval might be (0.5, 1.5], and the abnormal state support interval might be (1.5, +∞). Then, for each parameter's current measured value, based on its position within the preset confidence interval, its relationship to "normal" (…) is calculated. ),"fluctuation"( ),"abnormal"( These three single states and all their possible combinations (such as...) , , , The support share, also known as the basic probability allocation (BPA) or mass function m(·), is used to determine the probability distribution of a fluctuation. For example, if the slope of the radar residual envelope is 1.2, it falls within the support interval (0.5, 1.5] of the fluctuation state, but is farther from the boundary of the normal interval (0.5) and closer to the boundary of the abnormal interval (1.5). In this case, a higher support share can be assigned to the "fluctuation" state. At the same time, due to its abnormal bias, a portion of the support share was allocated to ,like The remaining support shares are allocated to the entire set. As an uncertainty, such as The same processing was applied to the magnitude of the displacement frequency decrease and the variance of the seat belt tension to obtain their respective BPA functions. and Ensure that the sum of all support shares for each BPA function is 1, and use these three BPA functions as the basic allocation parameters for heterogeneous probabilities.

[0034] The steps for obtaining the occupant disability probability assessment results are as follows: The trust intervals corresponding to the support shares for normal state, fluctuating state, and abnormal state in the heterogeneous probability basic allocation parameters are read item by item. The intersection range and conflict range of each trust interval are compared in turn. The support shares corresponding to the intersection range are superimposed, and the support shares corresponding to the conflict range are deducted. The deducted support shares are redistributed to the reserved state to form the fused abnormal state support shares. The probability value of the occupant abnormal behavior sudden state assessment is calculated. Then, the probability value of the occupant abnormal behavior sudden state assessment is compared with the preset disability probability judgment threshold item by item to determine whether the probability value of the occupant abnormal behavior sudden state assessment reaches the preset disability probability judgment threshold, and the occupant disability probability assessment result is obtained.

[0035] Specifically, the heterogeneous probability basic allocation parameters formed in the previous process are read item by item, namely the three independent BPA functions. The three sources of evidence are fused using the Dempster-Shafer Theory rule. First, the fusion... and Its combination formula is Where A and B are any focal elements (sets of states) with non-zero support in two BPA functions, C is their intersection, and K is the conflict coefficient, calculated using the following formula: This process superimposes the consistent supporting shares from the two sources of evidence and uses a normalization factor. The conflicting support shares are proportionally redistributed to the non-conflicting focal elements, resulting in a new, fused BPA function. Next, this new BPA function BPA function with the third source of evidence The Dempster-Shafer Theory rules are applied again for fusion to obtain the final fused BPA function. Extract the support share for the "abnormal" state from this final BPA function. The probability value for assessing the sudden state of abnormal occupant behavior is then compared with a preset disability probability judgment threshold. This threshold is the optimal balance point found through ROC curve analysis on a large amount of labeled data, taking into account both the false negative rate and the false positive rate. For example, the threshold can be set to 0.75. If the calculated probability value for assessing the sudden state of abnormal occupant behavior is greater than or equal to 0.75, the occupant is judged to be disabled and a high-risk alarm is output. Otherwise, the occupant's condition is considered to be within an acceptable range, and the final occupant disability probability assessment result is obtained.

Claims

1. A real-time monitoring system for the status of intelligent driving occupants based on multimodal data fusion, characterized in that, The system includes: The multimodal data extraction module obtains the pressure distribution values ​​of the seat contact surface, calculates and generates the two-dimensional center coordinates of the contact surface pressure, obtains the upper body point cloud dataset, and calculates and obtains the three-dimensional centroid coordinates of the occupant's torso. The occupant posture fusion module sets the three-dimensional centroid coordinates of the occupant's torso as a state variable parameter, calls the upper body tilt angle value and ischial tuberosity pressure transfer parameter to construct a spinal kinematic constraint algebraic expression, calculates and generates the posture residual covariance matrix, sets the two-dimensional center coordinates of the contact surface pressure as an observation variable parameter, and uses the observation variable parameter to calculate the error covariance gain weight addition of the posture residual covariance matrix to obtain the occupant's departure posture angle parameter; The vibration feature extraction module receives the echo continuous wave dataset, performs a two-dimensional frequency domain matrix transformation operation on the echo continuous wave dataset to obtain the distance Doppler two-dimensional frequency domain matrix, generates the chassis common mode vibration cancellation parameter, and extracts the residual signal dataset of the chassis common mode vibration cancellation parameter associated with the chest cavity boundary position through the occupant departure attitude angle parameter to obtain the occupant chest cavity displacement frequency feature parameter. The abnormal state assessment module calculates and generates heterogeneous probability basic allocation parameters based on the occupant's chest cavity displacement frequency characteristic parameters, performs calculations on the confidence intervals of each item of the heterogeneous probability basic allocation parameters, and obtains the occupant disability probability assessment results.

2. The intelligent driving occupant status real-time monitoring system based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the three-dimensional centroid coordinates of the occupant's torso are as follows: The pressure distribution values ​​of each sampling unit's contact surface output by the seat pressure sensor array are called. The pressure coordinate correspondence is established according to the lateral and longitudinal positions of each sampling unit in the seat plane. The pressure distribution values ​​of each sampling unit's contact surface are substituted into the origin offset calculation of each order of spatial geometric moment coordinate system. The lateral offset and longitudinal offset are counted. The two-dimensional center position offset parameter of the occupant's contact surface is calculated, and the two-dimensional center coordinate of the contact surface pressure is generated. Read all point coordinates in the upper body point cloud dataset, read the head node coordinates and torso skeleton node coordinates in the occupant biomechanical topology wireframe coordinate set, calculate the nearest neighbor distance for each point coordinate, correct the spatial correspondence between point coordinates and node coordinates according to the minimum correspondence of the nearest neighbor distance, extract the registration iteration association error value, and perform three-dimensional spatial translation vector transformation in combination with the two-dimensional center coordinates of the contact surface pressure to form the position values ​​of the occupant head node and the three-dimensional mesh node of the torso. Based on the position values ​​of the occupant's head node and torso 3D mesh node, the occupant's head node position value is removed, and the position values ​​of the chest node, abdomen node, shoulder node, and spine node in the torso 3D mesh node position value are retained. The three-axis coordinate reference of each torso 3D mesh node position value is unified, and the position components of each torso 3D mesh node position value on the three coordinate axes are accumulated. The average value of the position components on the three coordinate axes is calculated to obtain the occupant's torso 3D centroid coordinates.

3. The intelligent driving occupant status real-time monitoring system based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the attitude residual covariance matrix are as follows: The three-dimensional centroid coordinates of the occupant's torso are set as state variable parameters. The coordinate components of the three-dimensional centroid coordinates of the occupant's torso in the anterior-posterior, left-right, and vertical directions are extracted. The forward flexion angle, lateral tilt angle, and backward tilt angle components in the upper body tilt angle values ​​are read. The left transfer component, right transfer component, and forward transfer component in the ischial tuberosity pressure transfer parameters are read. The spinal kinematic constraint algebraic formula is written according to the displacement relationship of the spinal segment corresponding to each coordinate component. The spinal kinematic constraint algebraic formula is substituted into the state transition equation term by term. The coordinate changes of the state variable parameters at continuous time are recursively expanded. The component differences between the current state variable parameters and the recursive results are compared. The offset and offset direction of each component difference are statistically analyzed to obtain the prior estimation error of the state variable parameters. The prior estimation error of the state variable parameters is decomposed into error components in the front-back, left-right, and vertical directions. The fluctuation amplitude, number of changes, and offset concentration interval of each error component at continuous sampling time are statistically analyzed. The component correlation relationship is established according to the corresponding position of the error components in the same direction. The synchronous offset relationship between error components in different directions is calculated. The dispersion of each error component in each direction is written into the diagonal position, and the synchronous offset relationship of error components in different directions is written into the non-diagonal position to form the attitude residual covariance matrix.

4. The real-time monitoring system for intelligent driving occupant status based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the occupant dismount posture angle parameters are as follows: The two-dimensional center coordinates of the contact surface pressure are set as observation variable parameters. The lateral center coordinate components and longitudinal center coordinate components in the two-dimensional center coordinates of the contact surface pressure are extracted. The corresponding offset relationships between the lateral center coordinate components, longitudinal center coordinate components and the error components in each direction of the attitude residual covariance matrix are compared. The weight of the corresponding matrix component is increased according to the position of consistent offset, and the weight of the corresponding matrix component is compressed according to the position of opposite offset. The error covariance gain weight addition is calculated. The attitude components in each direction at the current moment are corrected according to the error covariance gain weight, and the occupant spatial attitude angle at the current moment is converted to obtain the occupant departure attitude angle parameters.

5. The intelligent driving occupant status real-time monitoring system based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the common-mode vibration cancellation parameters of the chassis are as follows: The echo continuous wave dataset is received by the vehicle-mounted millimeter-wave radar. The echo amplitude of each frame is sorted according to the range sampling order and velocity sampling order of the echo continuous wave dataset. The echo change corresponding to each range unit and the echo change corresponding to each velocity unit are extracted in sequence. The echo change of each frame is mapped to the intersection of the range dimension and the velocity dimension. The frequency domain distribution values ​​corresponding to the range position and the velocity position are written frame by frame to form a matrix arrangement result with a one-to-one correspondence between the range position and the velocity position, and the range-Doppler two-dimensional frequency domain matrix is ​​obtained. The longitudinal, lateral, and vertical acceleration signal values ​​are collected from the triaxial acceleration signal values ​​by the chassis inertial measurement unit. The longitudinal, lateral, and vertical acceleration signal values ​​are mapped to the same frame position in the range-Doppler two-dimensional frequency domain matrix. The vibration disturbance direction is calculated for each range unit and the vibration disturbance amplitude is calculated for each velocity unit. The radar echo common-mode vibration component values ​​in each matrix position are canceled frame by frame. The chest cavity reflection change component not covered by the longitudinal, lateral, and vertical acceleration signal values ​​is retained to generate chassis common-mode vibration cancellation parameters.

6. The real-time monitoring system for intelligent driving occupant status based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the occupant's thoracic cavity displacement frequency characteristic parameters are as follows: The flexion, roll, and rotation angles from the occupant displacement posture angle parameters are called. The projection positions of the thoracic cavity boundary in the distance and angle directions are corrected according to the flexion, roll, and rotation angles. The residual change values ​​of the corresponding positions of the thoracic cavity boundary are extracted frame by frame from the chassis common-mode vibration cancellation parameters to form a thoracic cavity boundary position residual signal sequence. The periodic displacement fluctuations in the thoracic cavity boundary position residual signal sequence are statistically analyzed in chronological order. The frequency positions with the most concentrated amplitude changes and the frequency positions with the most continuous frequency changes in each continuous time period are extracted to obtain the occupant thoracic cavity displacement frequency characteristic parameters.

7. The intelligent driving occupant status real-time monitoring system based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the basic allocation parameter of the heterogeneous probability are as follows: Based on the occupant's chest cavity displacement frequency characteristic parameters, the frequency position values ​​and spectral amplitude values ​​corresponding to continuous sampling times are extracted. The spectral amplitude values ​​are connected in chronological order to form a radar residual envelope. The envelope change and time interval of adjacent sampling times are statistically analyzed segment by segment. The slope change value corresponding to the direction of envelope change in each segment is calculated and summarized to form the slope value of the radar residual envelope. At the same time, the frequency position values ​​of adjacent sampling times are compared, the change difference of the frequency position value in the continuously decreasing segment is statistically analyzed and accumulated to form the displacement frequency decrease amplitude value. Then, the continuous seat belt tension value output by the seat belt Hall sensor is called, and the dispersion of the continuous seat belt tension value relative to the average tension value is calculated to form the seat belt tension variance value, thus obtaining a heterogeneous parameter set. Based on the heterogeneous parameter set, the values ​​of radar residual envelope slope, displacement frequency decrease amplitude, and seat belt tension variance are read respectively. According to the position of each value in the corresponding confidence interval, the support intervals for normal state, fluctuation state, and abnormal state are divided item by item. The interval proportion and boundary offset of each value falling into each confidence interval are calculated. The interval proportion and boundary offset are written into the support share of the corresponding state, the conflicting support shares are compressed, and the total benchmark of the support shares of each state is unified to form the basic allocation parameter of heterogeneous probability.

8. The real-time monitoring system for intelligent driving occupant status based on multimodal data fusion according to claim 1, characterized in that, The steps for obtaining the occupant disability probability assessment results are as follows: The trust intervals corresponding to the support shares for normal state, fluctuation state, and abnormal state in the heterogeneous probability basic allocation parameters are read item by item. The intersection range and conflict range of each trust interval are compared in turn. The support shares corresponding to the intersection range are superimposed, and the support shares corresponding to the conflict range are deducted. The deducted support shares are redistributed to the reserved state to form the fused abnormal state support shares. The occupant abnormal behavior sudden state assessment probability value is calculated. Then, the occupant abnormal behavior sudden state assessment probability value is compared with the preset disability probability judgment threshold item by item to determine whether the occupant abnormal behavior sudden state assessment probability value reaches the preset disability probability judgment threshold, and the occupant disability probability assessment result is obtained.

Images