A method for processing of driver multi-modal physiological data

Abstract

A method for processing multimodal physiological data of drivers is disclosed, belonging to the field of data intelligence processing technology. It solves the problem of data confidence assessment in the process of acquiring multi-source physiological signals, which is difficult to address in current data processing solutions. Specifically, the method includes: multimodal data acquisition: acquiring multimodal data related to driver state information; data synchronization processing: performing data synchronization processing on the acquired multimodal data to obtain synchronized data; multimodal feature acquisition: obtaining driver state information from the synchronized data; confidence calculation: determining the confidence level of the driver state information; and structured prompt word generation: constructing structured prompt words to assist in large-scale model judgment based on the driver state information and its confidence level.

Description

Technical Field

[0001] This invention belongs to the field of data intelligence processing technology, specifically relating to a method for processing multimodal physiological data of drivers. Background Technology

[0002] In intelligent monitoring scenarios based on digital data processing, the accuracy and reliability of data processing are crucial. Among these, the acquisition, processing, and intelligent reasoning of human physiological signals is one of the important application directions of digital data processing technology. In recent years, a large number of status assessment technologies based on the acquisition and processing of human physiological signals have emerged. By digitally acquiring, preprocessing, and analyzing physiological signals, digital information is generated to assess the status of the target object. Then, when the assessment results do not meet preset safety requirements, relevant data output operations such as prompts or alarms are triggered.

[0003] In the process of digitally analyzing and evaluating human physiological state information using large-scale language reasoning models, the structured text descriptions (prompt words) input into the large-scale model are a crucial prerequisite for accurate reasoning operations, directly affecting the accuracy and reliability of data processing. Current technologies, focusing on the core data processing step of constructing prompt words from collected physiological information into large-scale model prompts, emphasize building richer prompt content. By increasing the information dimensions of the prompt words, the accuracy of the large-scale model's reasoning operations is improved. For example, the Chinese invention patent application "A Large-Scale Model-Driven Human-like Evaluation Method for Intelligent Vehicle Test Scenarios" (Publication No. CN 120145204 A) constructs a combined dataset including "prompt words - chassis dynamics - physiological data - evaluation terms," ​​optimizing the large-scale model's reasoning performance and improving the accuracy of digital data processing by enriching the information dimensions of the prompt content.

[0004] However, in the process of synchronous digital acquisition and processing of multiple physiological signals, problems such as data asynchrony, data loss, and data errors inevitably arise due to factors such as differences in the timing of multi-source data acquisition, transmission interference, and the accuracy of acquisition equipment, affecting data confidence. Currently, in the field of physiological data processing, there is a lack of relevant technical research on how to optimize prompt word data through data confidence judgment to obtain more accurate input data for large models. Existing data processing solutions are unable to solve the problem of data confidence assessment in the process of multi-source physiological signal acquisition, resulting in accuracy defects in the prompt words input to large models, affecting the accuracy of large model inference operations, and restricting the application effect of physiological signal data processing technology based on large models. Summary of the Invention

[0005] To address the technical problem that current data processing solutions struggle with data confidence assessment during multi-source physiological signal acquisition, leading to accuracy deficiencies in input prompts for large models, this invention provides a method for processing multimodal physiological data of drivers. The method specifically comprises: S1. Multimodal data acquisition: Acquiring multimodal data related to driver status information; S2. Data synchronization processing: Perform data synchronization processing on the collected multimodal data to obtain synchronized data; S3. Multimodal feature acquisition: Obtain driver status information from synchronized data; S4. Confidence Calculation: Determine the confidence level of the driver's status information; S5. Structured prompt word generation: Construct structured prompt words to assist the large model in making decisions based on driver status information and the confidence level of the driver status information.

[0006] Furthermore, multimodal data acquisition obtains multimodal data related to driver information by subscribing to ROS2 topics, including driver facial image data and driver seat pressure distribution data.

[0007] Furthermore, the driver's facial image data is collected by using an in-vehicle camera to capture the driver's facial image, and the collected data is in the form of continuous video frame data; the driver's seat pressure distribution data is collected by placing pressure sensors on the driver's seat cushion and backrest to obtain the seat pressure distribution.

[0008] Furthermore, the synchronization process employs a sliding time window, specifically: Timestamp standardization: Each video frame and each stress data point is stamped with a millisecond-level system timestamp to unify the time base; Window parameters are defined: The window length and sliding step size are set, and the rule is that data timestamps falling within the window range are included in the window; Sliding window filtering: Starting from the earliest data timestamp, the window slides according to the step size to filter all video frames and stress data within the window time range and integrate them into a synchronous data unit.

[0009] Furthermore, obtaining driver status information from synchronized data specifically involves: extracting the driver's attention region from driver face image video frame data and calculating the driver's heart rate variability from driver face image video frame data; and extracting driver seat pressure distribution features from driver seat pressure distribution data. Extracting the driver's attention region from driver face image video frame data specifically involves: given a driver's face image, using a CNN / Transformer network to classify the driver's attention region. The classification result is determined by the given training labels, including the front, left rearview mirror, right rearview mirror, center console screen, dashboard, and others. The calculation of driver heart rate variability from driver facial image video frame data specifically involves: using camera-based non-contact heart rate detection technology to detect driver heart rate variability data HRV; Extracting driver seat pressure distribution features from driver seat pressure distribution data involves converting the driver seat pressure distribution data into an M*N matrix, which is divided into four parts: left side of the seat cushion, right side of the seat cushion, left side of the backrest, and right side of the backrest. The driver seat pressure distribution features are obtained by analyzing the pressure conditions of these four parts.

[0010] Furthermore, the confidence levels of driver status information include confidence levels for attention information, heart rate information, and stress information, all uniformly adopting... The calculation method, in which Indicates the sliding window index. Indicates the confidence score; The score represents the number of samples. ,but ,like ,but ,in For the sample size, For the target sample size, The minimum number of samples to be set; This indicates the time consistency score. , This indicates the timestamp span of the samples selected within the sliding window. The length of the sliding window; The signal quality score is represented by setting the signal quality scores for attention information confidence and stress information confidence to 1, calculating the signal quality score for heart rate information confidence separately, and defining the baseline signal range. , The minimum threshold for heart rate variability (HRV) data. The highest threshold for heart rate variability (HRV) data was selected. For each heart rate variability (HRV) sample z, calculate its single-sample out-of-bounds penalty coefficient. , The constant is used to prevent division by zero; calculate the average out-of-bounds penalty coefficient within the sliding window: , .

[0011] Furthermore, during the construction of structured prompts, based on driver state information and the confidence level of that information, structured prompts are constructed to assist in the large model's judgment. The specific implementation steps are as follows: S51. When the confidence level of a certain status information is greater than or equal to the first threshold, the status information is determined to be credible and can be written into structured prompt words as a basis for normal evaluation. S52. When the confidence level of the state information is less than the first threshold but greater than or equal to the second threshold, the confidence level of the state information is deemed insufficient. The state information shall only be written into the structured prompt words as auxiliary reference information and shall not be used as the basis for triggering the strong constraint evaluation rule. S53. When the confidence level of the status information is less than the second threshold but greater than or equal to the third threshold, the reliability of the status information is determined to be low, and the weight of the status information on the driver's status assessment is reduced to a preset low weight. S54. When the confidence level of a certain status information is less than the third threshold, the status information is determined to be invalid, marked as invalid information in the structured prompt words, and will not participate in the final evaluation.

[0012] The beneficial effects of the method described in this invention are as follows: To address the issues in existing multimodal physiological data processing where data asynchrony, missing information, and errors due to factors such as differences in the timing of multi-source data acquisition, transmission interference, and equipment precision lead to insufficient accuracy of input prompts and limited inference accuracy in large models, this invention combines multimodal data related to driver state. Through a systematic process of data acquisition, synchronous processing, feature extraction, confidence calculation, and structured prompt construction, it deeply integrates confidence assessment results with driver state information to construct structured prompts. This guides large models to perform inferences based on data reliability, avoiding interference from low-confidence data. In the confidence calculation stage, a comprehensive judgment is made based on multiple dimensions of multimodal data, such as sample size, time consistency, and signal stability, to avoid confidence bias caused by evaluation of a single indicator. Through the combination of these solutions, driver multimodal physiological data processing achieves high accuracy and robustness. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method described in an embodiment of the present invention. Detailed Implementation

[0014] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0015] Example 1 This embodiment provides a method for processing multimodal physiological data of drivers, such as... Figure 1 The flowchart of the method is shown below: Step 1: Multimodal data acquisition: Multimodal data related to driver information, including driver facial image data and driver seat pressure distribution data, is acquired by subscribing to ROS2 topics. The collected multimodal data related to driver information is then input into the data synchronization and preprocessing module.

[0016] The driver's facial image data is collected by using an in-vehicle camera to capture the driver's facial image, and the collected data is in the form of continuous video frames.

[0017] The driver's seat pressure distribution data is collected by placing pressure sensors on the driver's seat to obtain the seat pressure distribution. In this embodiment, the SensAxis flexible surface pressure distribution testing system is used to obtain the driver's seat pressure distribution data.

[0018] Step 2, Data Synchronization Processing: The data to be processed consists of driver facial images input as continuous video frames and pressure distribution data input as continuous data. These two data need to be processed synchronously using a sliding time window, specifically: Timestamp standardization: Each video frame and each stress data point is stamped with a millisecond-level system timestamp to unify the time base; Window parameters are defined: The window length and sliding step are set, and the rule is that data timestamps falling within the window range are included in the window, adapting to millisecond-level precision; Sliding window filtering: Starting from the earliest data timestamp, the window slides according to the step size to filter all video frames and stress data within the window time range and integrate them into a synchronous data unit.

[0019] After data synchronization processing, multimodal feature extraction and confidence calculation are performed in the form of continuous synchronized data units. A data unit is all the data within a set sliding time window.

[0020] Step 3: Multimodal feature extraction: Multimodal feature extraction includes extracting the driver's attention region from driver face image video frame data and calculating the driver's heart rate variability from driver face image video frame data; and extracting driver seat pressure distribution features from driver seat pressure distribution data.

[0021] Extracting driver attention regions from driver face image video frame data specifically involves: given a driver face image, using an existing CNN / Transformer network to classify driver attention regions. The classification result is determined by the labels in the given training set; in this embodiment, the labels are set to "front," "left rearview mirror," "right rearview mirror," "center console screen," "instrument panel," and "other." Through deep learning, the image data captured by the camera is mapped one by one to the different predefined attention classification regions, transforming facial image data over a period of time into attention region data.

[0022] The results obtained from consecutive time frames are processed to form a one-to-one mapping. The driver's facial information at a certain moment is directly mapped to the area the driver is currently looking at. This includes "Ahead," "Left Rearview Mirror," "Right Rearview Mirror," "Central Control Screen," "Instrument Panel," and "Other."

[0023] The calculation of driver heart rate variability from driver facial image video frame data specifically involves: using camera-based contactless heart rate detection technology (rPPG), an in-vehicle camera continuously acquires driver facial image video frames, selecting exposed skin areas such as the forehead and cheeks as target areas, and statistically analyzing the color and brightness changes of these areas in consecutive frames; since heartbeats cause blood to flow periodically under the skin, resulting in slight periodic changes in skin color over time, this change signal is filtered, denoised, and frequency analyzed to extract the dominant frequency component corresponding to the heartbeat rhythm, which is then converted into the driver's heart rate, enabling continuous monitoring of heart rate and thus obtaining driver heart rate variability data (HRV).

[0024] Extracting driver seat pressure distribution features from driver seat pressure distribution data involves converting the driver seat pressure distribution data into an M*N matrix, which is divided into four parts: left side of the seat cushion, right side of the seat cushion, left side of the backrest, and right side of the backrest. The driver seat pressure distribution features are obtained by analyzing the pressure conditions of these four parts.

[0025] Data semantic classification: Left side of the seat cushion: SL = mat[:3, :8] (rows 0~2, columns 0~7); Right side of the seat cushion: SR = mat[3:, :8] (rows 3~5, columns 0~7); Left side of the backrest: BL = mat[:3, 8:] (rows 0~2, columns 8~17); Right side of the backrest: BR = mat[3:, 8:] (rows 3~5, columns 8~17); Target quantity, regional average pressure: ; ; ; ; This indicates the average pressure on the right side of the seat cushion. This indicates the number of time points within a time window. Indicates the first The average value of all values ​​in the matrix (SR) on the right side of the seat cushion at each time point.

[0026] This indicates the average pressure on the left side of the seat cushion. Indicates the first The average value of all values ​​in the left side matrix (SL) of the seat cushion at each time point.

[0027] This indicates the average pressure on the left side of the backrest. Indicates the first The average value of all values ​​in the left side matrix (BL) of the backrest at each time point.

[0028] This indicates the average pressure on the right side of the seat back. Indicates the first The average value of all values ​​in the right side matrix (BR) of the backrest at each time point.

[0029] Similarly, we can obtain , Calculate the average of the seat cushion matrix and the backrest matrix.

[0030] Calculate the standard deviation of pressure fluctuations: ; ; This indicates pressure fluctuations in the seat cushion area. Indicates the first The standard deviation of the pressure data of the seat cushion area at each time point This indicates pressure fluctuations in the backrest area. Indicates the first The standard deviation of the backrest area pressure data at each time point.

[0031] Symmetry index, absolute left-right difference:

[0032]

[0033] in, This indicates the absolute left-right difference of the seat cushion. This indicates the absolute left-right difference of the backrest; Symmetry index, left-right difference ratio: ; ; in, It is a constant, the purpose of which is to prevent division by zero; This indicates the left-right difference ratio of the seat cushion's symmetry. This indicates the left-right difference ratio of the backrest symmetry.

[0034] Coefficient of variation: ; ; in, This represents the coefficient of variation of the backrest. This represents the coefficient of variation of the seat cushion.

[0035] Backrest support trend: obtained by pressure changes within different time windows.

[0036] Step S4, Confidence Calculation: Confidence calculation includes calculating confidence scores for attention information, heart rate information, and stress information.

[0037] The confidence levels for driver status information include confidence levels for attention information, heart rate information, and stress information, and are uniformly adopted. The calculation method, in which Indicates the sliding window index. Indicates the confidence score; Sample size score : like If the modality is determined to be "insufficient" in the window, then directly... , like Then the quantity will be mapped to And in Saturation point: Where N is the number of samples, For the target sample size, The minimum number of samples to be set; Time Consistency Score : ; in, This represents the timestamp span of the samples selected within the sliding window, while The window length is defined directly.

[0038] when The samples were concentrated in a very short period of time ( This indicates a high degree of consistency in time. When the sample is dispersed and covers the entire window ( They believe that time consistency is low. .

[0039] Signal quality score : The signal quality scores for attention information confidence and stress information confidence were set to 1, and the signal quality score for heart rate information confidence was calculated separately. Define the basic signal range , The minimum threshold for heart rate variability (HRV) data. The maximum threshold for heart rate variability (HRV) data is based on a medically established normal physiological range for HRV. Values ​​exceeding this range are considered abnormal values ​​affected by noise, motion artifacts, or signal dropout.

[0040] For the samples involved in the calculation within the sliding window, select several HRV data samples z and calculate their single-sample out-of-bounds penalty coefficient. Its value is fixed in the range [0,1]. , It is a constant, the purpose of which is to prevent division by zero; Average out-of-bounds penalty within the window: , .

[0041] in E is the total number of samples within the sliding window, and E is the average penalty value of all samples within the window, representing the average degree of out-of-bounds violation for the entire window, with a value range of [0,1]. The larger E is, the more abnormal samples there are within the window, the more severe the out-of-bounds violation, and the worse the signal quality.

[0042] The mainstream techniques commonly used for confidence assessment mostly employ threshold discrimination, machine learning and deep learning methods, and multi-dimensional fusion methods.

[0043] Traditional threshold discrimination methods directly filter out data that is outside the acceptable range, failing to distinguish between "minor violations" and "serious violations." They also cannot accurately reflect the degree of signal loss. The method proposed in this invention incorporates the concept of the degree of violation, resulting in more reliable confidence. Furthermore, it is computationally simple, with low computational and memory overhead, making it ideal for real-time health monitoring. It also offers strong interpretability and eliminates black-box operations.

[0044] Step S5: Generation of structured prompt words: Structured prompt word generation utilizes six types of data obtained from multimodal feature extraction and confidence calculation: driver attention region, driver heart rate variability, driver seat pressure features, attention information confidence, heart rate information confidence, and pressure information confidence to construct structured prompt words. During structured prompt word construction, based on driver state information and its confidence, structured prompt words are built to assist in the large model's judgment. The specific implementation steps are as follows: S51. When the confidence level of a certain status information is greater than or equal to the first threshold, the status information is determined to be credible and can be written into the structured prompt words as a basis for normal judgment. S52. When the confidence level of the state information is less than the first threshold but greater than or equal to the second threshold, the confidence level of the state information is deemed insufficient. The state information shall only be written into the structured prompt words as auxiliary reference information and shall not be used as the basis for triggering the strong constraint evaluation rule. S53. When the confidence level of the status information is less than the second threshold but greater than or equal to the third threshold, the reliability of the status information is determined to be low, and the weight of the status information on the driver's status assessment is reduced to a preset low weight. S54. When the confidence level of a certain status information is less than the third threshold, the status information is determined to be invalid, marked as invalid information in the structured prompt words, and will not participate in the final evaluation.

[0045] The first threshold is set to 0.70, the second threshold is set to 0.30, and the third threshold is set to 0.10. When the confidence level of a certain modality is less than 0.30, its influence weight is adjusted to 10%; when the confidence level of a certain modality is less than 0.10, its influence weight is adjusted to 0.

Claims

1. A method for processing multimodal physiological data of drivers, characterized in that, The method is specifically as follows: S1. Multimodal data acquisition: Acquiring multimodal data related to driver status information; S2. Data synchronization processing: Perform data synchronization processing on the collected multimodal data to obtain synchronized data; S3. Multimodal feature acquisition: Obtain driver status information from synchronized data; S4. Confidence Calculation: Determine the confidence level of the driver's status information; S5. Structured prompt word generation: Construct structured prompt words to assist the large model in making decisions based on driver status information and the confidence level of the driver status information.

2. The method for processing multimodal physiological data of drivers according to claim 1, characterized in that, Multimodal data acquisition involves subscribing to ROS2 topics to obtain multimodal data related to driver information, including driver facial image data and driver seat pressure distribution data.

3. The method for processing multimodal physiological data of drivers according to claim 2, characterized in that, The driver's facial image data is collected by using an in-vehicle camera to capture the driver's facial image, and the collected data is in the form of continuous video frames; the driver's seat pressure distribution data is collected by placing pressure sensors on the driver's seat cushion and backrest to obtain the seat pressure distribution.

4. The method for processing multimodal physiological data of drivers according to claim 3, characterized in that, The synchronization process uses a sliding time window, specifically: Timestamp standardization: Each video frame and each stress data point is stamped with a millisecond-level system timestamp to unify the time base; Window parameters are defined: The window length and sliding step size are set, and the rule is that data timestamps falling within the window range are included in the window; Sliding window filtering: Starting from the earliest data timestamp, the window slides according to the step size to filter all video frames and stress data within the window time range and integrate them into a synchronous data unit.

5. A method for processing multimodal physiological data of drivers according to claim 4, characterized in that, The specific steps for obtaining driver status information from synchronized data are as follows: extracting the driver's attention region from the driver's facial image video frame data and calculating the driver's heart rate variability from the driver's facial image video frame data; and extracting the driver's seat pressure distribution characteristics from the driver's seat pressure distribution data. Extracting the driver's attention region from driver face image video frame data specifically involves: given a driver's face image, using a CNN / Transformer network to classify the driver's attention region. The classification result is determined by the given training labels, including the front, left rearview mirror, right rearview mirror, center console screen, dashboard, and others. The calculation of driver heart rate variability from driver facial image video frame data specifically involves: using camera-based non-contact heart rate detection technology to detect driver heart rate variability data HRV; Extracting driver seat pressure distribution features from driver seat pressure distribution data involves converting the driver seat pressure distribution data into an M*N matrix, which is divided into four parts: left side of the seat cushion, right side of the seat cushion, left side of the backrest, and right side of the backrest. The driver seat pressure distribution features are obtained by analyzing the pressure conditions of these four parts.

6. The method for processing multimodal physiological data of drivers according to claim 5, characterized in that, The confidence levels for driver status information include confidence levels for attention information, heart rate information, and stress information, and are uniformly adopted. The calculation method, in which Indicates the sliding window index. Indicates the confidence score; The score represents the number of samples. ,but ,like ,but ,in For the sample size, For the target sample size, The minimum number of samples to be set; This indicates the time consistency score. , This indicates the timestamp span of the samples selected within the sliding window. The length of the sliding window; The signal quality score is represented by setting the signal quality scores for attention information confidence and stress information confidence to 1, calculating the signal quality score for heart rate information confidence separately, and defining the baseline signal range. , The minimum threshold for heart rate variability (HRV) data. The highest threshold for heart rate variability (HRV) data was selected. For each heart rate variability (HRV) sample z, calculate its single-sample out-of-bounds penalty coefficient. , The constant is used to prevent division by zero; calculate the average out-of-bounds penalty coefficient within the sliding window: , .

7. A method for processing multimodal physiological data of drivers according to claim 6, characterized in that, When constructing structured prompts, based on driver state information and the confidence level of that information, structured prompts are built to assist in the judgment of large models. The specific implementation steps are as follows: S51. When the confidence level of a certain status information is greater than or equal to the first threshold, the status information is determined to be credible and can be written into structured prompt words as a basis for normal evaluation. S52. When the confidence level of the state information is less than the first threshold but greater than or equal to the second threshold, the confidence level of the state information is deemed insufficient. The state information shall only be written into the structured prompt words as auxiliary reference information and shall not be used as the basis for triggering the strong constraint evaluation rule. S53. When the confidence level of the status information is less than the second threshold but greater than or equal to the third threshold, the reliability of the status information is determined to be low, and the weight of the status information on the driver's status assessment is reduced to a preset low weight. S54. When the confidence level of a certain status information is less than the third threshold, the status information is determined to be invalid, marked as invalid information in the structured prompt words, and will not participate in the final evaluation.

Images