A steering wheel hand-off detection method and device based on multi-modal information fusion
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- VOYAH AUTOMOBILE TECH CO LTD
- Filing Date
- 2026-02-27
- Publication Date
- 2026-06-19
Smart Images
Figure CN122241396A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent driving assistance systems for vehicles, and in particular to a method and device for detecting steering wheel hands-off detection based on multimodal information fusion. Background Technology
[0002] Currently, with the continuous improvement of automotive intelligence, intelligent driving assistance systems are widely used in vehicles. During the activation of assisted driving functions, the system needs to monitor in real time whether the driver's hands are on the steering wheel to ensure that the driver can take over the vehicle promptly in dangerous situations that the system cannot handle. Therefore, hands-on detection (HOD) has become one of the key functions of intelligent driving systems.
[0003] Currently, mainstream steering wheel hands-off detection technology is mainly based on the principle of capacitive sensing. By placing a capacitive sensor inside the steering wheel, it utilizes the characteristic that the driver's hand contact causes changes in the sensor's capacitance or oscillation frequency to determine whether the driver is holding the steering wheel. However, this type of technology has problems in practical applications.
[0004] On the one hand, the detection signal of capacitive sensors is easily affected by environmental factors. For example, when hot or cold air from a car's air conditioning system blows directly onto the steering wheel, it causes a rapid, non-linear drift in the sensor signal baseline. Changes in ambient temperature and humidity can also alter the sensor's medium properties, thus affecting the accuracy of the detection results. Traditional baseline tracking and threshold compensation algorithms suffer from response lag and struggle to accurately distinguish between hand contact and environmental interference in dynamically changing environments, leading to frequent false alarms or missed alarms.
[0005] On the other hand, existing technologies make relatively simple use of capacitive sensor signals, usually focusing only on macroscopic scalar characteristics such as instantaneous values or moving averages of the signal, ignoring the rich dynamic patterns contained in the signal in the time and frequency domain. Existing solutions fail to fully explore and utilize these deep information, limiting the further improvement of their anti-interference capabilities.
[0006] Therefore, there is an urgent need for a steering wheel hands-off detection method that can effectively suppress environmental interference, fully utilize the deep characteristics of capacitor frequency signals, and possess high reliability and strong anti-interference capabilities, in order to improve the safety and user experience of intelligent driving assistance systems. Summary of the Invention
[0007] In order to overcome the shortcomings of existing technologies, such as low detection reliability due to environmental interference and insufficient utilization of capacitance frequency signal characteristics, and to achieve high-precision and high-robustness steering wheel hands-off detection, this invention provides a steering wheel hands-off detection method and device based on multimodal information fusion.
[0008] In a first aspect, embodiments of the present invention provide a steering wheel hands-off detection method based on multimodal information fusion, which may include:
[0009] Acquire the environmental state vector and the capacitance frequency signal of the vehicle steering wheel; wherein the capacitance frequency signal is used to characterize the contact state between the vehicle steering wheel and the driver's hand; A time-frequency transformation is performed on the capacitor frequency signal to obtain a time-frequency spectrum. The time-spectrum diagram and the environmental state vector are input into a preset deep spectrum fusion model to obtain the first hand-drop detection result and the corresponding first confidence level. Timing features are extracted based on the capacitor frequency signal; The time-series features are input into a preset time-series feature model to obtain the second release detection result and the corresponding second confidence level. Based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level, the steering wheel hand-off detection result is obtained.
[0010] In one or more optional embodiments of this application, acquiring the environmental state vector and the capacitance frequency signal of the vehicle steering wheel includes: Based on the capacitive sensor of the vehicle steering wheel, the capacitive frequency signal is collected with the current sampling time as the endpoint and a preset time window is traced back. Collect an environmental state vector that traces back a preset time window from the current sampling time as the endpoint; wherein, the environmental state vector includes at least one of ambient temperature, air conditioning air volume, air conditioning set temperature, and air outlet direction.
[0011] In one or more optional embodiments of this application, the step of extracting timing features based on the capacitor frequency signal includes: The capacitor frequency signal is subjected to feature calculations to obtain the moving average sequence, standard deviation sequence, and gradient sequence of the capacitor frequency signal; The time series features are obtained by concatenating the moving average sequence, the standard deviation sequence, and the gradient sequence.
[0012] In one or more optional embodiments of this application, obtaining the steering wheel release detection result based on the first release detection result, the first confidence level, the second release detection result, and the second confidence level includes: If the first confidence level is greater than the second confidence level, then the first hand-off detection result is taken as the steering wheel hand-off detection result; If the second confidence level is greater than the first confidence level, then the second hand-off detection result is taken as the steering wheel hand-off detection result.
[0013] In one or more optional embodiments of this application, obtaining the steering wheel release detection result based on the first release detection result, the first confidence level, the second release detection result, and the second confidence level includes: Based on the historical accuracy of the preset deep spectrum fusion model and the preset temporal feature model under the working conditions corresponding to the environmental state vector, the first weight corresponding to the preset deep spectrum fusion model and the second weight corresponding to the preset temporal feature model are determined. Based on the first weight and the first confidence level, a first weighted confidence level is obtained; Based on the second weight and the second confidence level, a second weighted confidence level is obtained; If the first weighted confidence level is greater than the second weighted confidence level, then the first hands-off detection result is taken as the steering wheel hands-off detection result; If the second weighted confidence level is greater than the first weighted confidence level, then the second hands-off detection result is taken as the steering wheel hands-off detection result.
[0014] In one or more optional embodiments of this application, obtaining the steering wheel release detection result based on the first release detection result, the first confidence level, the second release detection result, and the second confidence level includes: If the first hand-off detection result and the second hand-off detection result are the same, then the first hand-off detection result or the second hand-off detection result shall be taken as the steering wheel hand-off detection result; If the first hand-off detection result and the second hand-off detection result are different, the steering wheel hand-off detection result is determined based on the historical steering wheel hand-off detection results and the environmental state vector.
[0015] In one or more optional embodiments of this application, the step of determining the steering wheel hand-off detection result based on historical steering wheel hand-off detection results and environmental information if the first hand-off detection result and the second hand-off detection result are different includes: If the historical steering wheel hands-off detection results remain stable within a preset time period, and the environmental state vector indicates that the vehicle is not in a preset high-interference condition, then the historical steering wheel hands-off detection results will be used as the steering wheel hands-off detection results. If the environmental state vector indicates that the vehicle is in a preset high-interference condition, then the steering wheel hands-off detection result is determined from the first hands-off detection result and the second hands-off detection result based on the first confidence level and the second confidence level.
[0016] In one or more optional embodiments of this application, before acquiring the environmental state vector and the capacitance frequency signal of the vehicle steering wheel, the method further includes: Under various operating conditions, acquire multiple historical environmental state vectors and multiple historical capacitance frequency signals collected by the capacitance sensor of the vehicle steering wheel; The preset deep spectrum fusion model is obtained by training the model based on the multiple historical environmental state vectors and the multiple historical capacitor frequency signals. The preset time-series feature model is obtained by training the model based on the multiple historical capacitor frequency signals.
[0017] Secondly, embodiments of the present invention provide a steering wheel hands-off detection device based on multimodal information fusion, which may include: The data acquisition module is used to acquire environmental state vectors and capacitance frequency signals of the vehicle steering wheel; wherein, the capacitance frequency signals are used to characterize the contact state between the vehicle steering wheel and the driver's hand. The time-frequency conversion module is used to perform time-frequency conversion based on the capacitor frequency signal to obtain a time-frequency spectrum. The first prediction module is used to input the time-spectrum map and the environmental state vector into a preset deep spectrum fusion model to obtain the first off-hand detection result and the corresponding first confidence level. The data extraction module is used to extract time-series features based on the capacitor frequency signal; The second prediction module is used to input the time series features into a preset time series feature model to obtain the second release detection result and the corresponding second confidence level. The result determination module is used to obtain the steering wheel hand-off detection result based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level.
[0018] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the steering wheel hands-off detection method based on multimodal information fusion as described above.
[0019] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steering wheel hands-off detection method based on multimodal information fusion as described above.
[0020] Fifthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steering wheel hands-off detection method based on multimodal information fusion as described above.
[0021] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: This invention provides a steering wheel hands-off detection method based on multimodal information fusion. This method acquires an environmental state vector and a capacitance frequency signal characterizing the contact state between the steering wheel and the hand, laying the data foundation for subsequent multimodal fusion. Based on this, on one hand, the capacitance frequency signal undergoes time-frequency transformation to obtain a time-spectrum graph, which is then input together with the environmental state vector into a deep spectrum fusion model. This achieves the extraction of deep signal features from the frequency domain dimension and combines them with environmental information for comprehensive judgment, resulting in a first hands-off detection result that effectively distinguishes between hand contact and environmental interference. On the other hand, temporal features are extracted from the capacitance frequency signal and input into a temporal feature model to obtain a second hands-off detection result, capturing the dynamic change pattern of the capacitance frequency signal from the time domain dimension, forming a complementary advantage with the spectrum model. Finally, by fusing the two hands-off detection results and their confidence levels for comprehensive decision-making, a steering wheel hands-off detection result is obtained, significantly improving the accuracy and reliability of steering wheel hands-off detection. This method fully utilizes the frequency domain features, temporal features, and environmental state vector of the capacitance frequency signal to achieve multi-dimensional information fusion, exhibiting strong anti-interference capabilities and scene adaptability.
[0022] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.
[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0024] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the steering wheel hands-off detection method based on multimodal information fusion provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the steering wheel hands-off detection device based on multimodal information fusion provided in an embodiment of the present invention. Detailed Implementation
[0025] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0026] The inventors discovered that current mainstream steering wheel hands-off detection technologies are primarily based on capacitive sensing principles. By placing a capacitive sensor inside the steering wheel, they utilize the characteristic that driver hand contact causes changes in the sensor's capacitance or oscillation frequency to determine whether the driver is holding the steering wheel. However, this type of technology has problems in practical applications.
[0027] On the one hand, the detection signal of capacitive sensors is easily affected by environmental factors. For example, when hot or cold air from a car's air conditioning system blows directly onto the steering wheel, it causes a rapid, non-linear drift in the sensor signal baseline. Changes in ambient temperature and humidity can also alter the sensor's medium properties, thus affecting the accuracy of the detection results. Traditional baseline tracking and threshold compensation algorithms suffer from response lag and struggle to accurately distinguish between hand contact and environmental interference in dynamically changing environments, leading to frequent false alarms or missed alarms.
[0028] On the other hand, existing technologies utilize capacitive sensor signals in a relatively simplistic way, typically focusing only on macroscopic scalar characteristics such as instantaneous values or moving averages, neglecting the rich dynamic modes inherent in the signal in the time and frequency domain. Existing solutions fail to fully explore and utilize this deeper information, limiting further improvements in their anti-interference capabilities. Based on this, the inventors, through further research and development, have created this invention, providing a steering wheel hands-off detection method and device based on multimodal information fusion.
[0029] Example 1 Embodiment 1 of the present invention provides a steering wheel hands-off detection method based on multimodal information fusion, referring to... Figure 1 As shown, the method may include the following steps S101-S106: S101: Acquire the environmental state vector and the capacitance frequency signal of the vehicle steering wheel. The capacitance frequency signal is used to characterize the contact state between the vehicle steering wheel and the driver's hands.
[0030] S102: Perform time-frequency transformation based on the capacitor frequency signal to obtain the time-frequency spectrum.
[0031] S103: Input the time spectrum and environmental state vector into the preset deep spectrum fusion model to obtain the first off-hand detection result and the corresponding first confidence level.
[0032] S104: Time-series features are extracted based on capacitor frequency signals.
[0033] S105: Input the temporal features into the preset temporal feature model to obtain the second release detection result and the corresponding second confidence level.
[0034] S106: Based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level, the steering wheel hand-off detection result is obtained.
[0035] This invention provides a steering wheel hands-off detection method based on multimodal information fusion. This method acquires an environmental state vector and a capacitance frequency signal characterizing the contact state between the steering wheel and the hand, laying the data foundation for subsequent multimodal fusion. Based on this, on one hand, the capacitance frequency signal undergoes time-frequency transformation to obtain a time-spectrum graph. This time-spectrum graph and the environmental state vector are then input into a deep spectrum fusion model, enabling the extraction of deep signal features from the frequency domain dimension and comprehensive judgment based on environmental information to obtain a first hands-off detection result, effectively distinguishing between hand contact and environmental interference. On the other hand, temporal features are extracted from the capacitance frequency signal and input into a temporal feature model to obtain a second hands-off detection result. This captures the dynamic change pattern of the capacitance frequency signal from the time domain dimension, forming a complementary advantage with the spectrum model. Finally, by fusing the two hands-off detection results and their confidence levels for comprehensive decision-making, a steering wheel hands-off detection result is obtained, significantly improving the accuracy and reliability of steering wheel hands-off detection. This method fully utilizes the frequency domain features, temporal features, and environmental state vector of the capacitance frequency signal to achieve multi-dimensional information fusion, exhibiting strong anti-interference capabilities and scene adaptability.
[0036] In step S101 above, the environmental state vector and the capacitance frequency signal of the vehicle steering wheel are acquired. The capacitance frequency signal is used to characterize the contact state between the vehicle steering wheel and the driver's hands. Specifically, this includes the following steps S1011-S1012: S1011: A capacitive sensor based on the vehicle steering wheel, which collects the capacitive frequency signal based on a preset extraction frequency, with the current sampling time as the endpoint and a preset time window preceding it.
[0037] Specifically, this can be achieved by using a capacitive sensor installed inside the vehicle's steering wheel to collect the capacitance frequency signal in real time. Considering the real-time and accuracy requirements of steering wheel hands-off detection, a preset extraction frequency of 10-50 Hz can be set, preferably 50 Hz, meaning the capacitance frequency signal is collected every 20 milliseconds.
[0038] Meanwhile, to ensure the stability and reliability of subsequent temporal feature extraction, each sampling moment does not collect a single instantaneous value, but rather a sequence of capacitance frequency signals that traces back a preset time window from the current sampling moment. The length of this preset time window can be set to 1-5 seconds, preferably 2 seconds, meaning that each sampling moment acquires the capacitance frequency signal sequence within the past 2 seconds.
[0039] S1012: Collect the environmental state vector, which is the current sampling time and traces back a preset time window. The environmental state vector includes at least one of the following: ambient temperature, air conditioning air volume, air conditioning set temperature, and air outlet direction.
[0040] Specifically, environmental state information can be acquired in real time via the vehicle's CAN bus or onboard sensor network. In conjunction with the acquisition of the capacitor frequency signal, the acquisition of the environmental state vector also uses the current sampling time as the endpoint and traces back the same preset time window to ensure that the environmental state vector and capacitor frequency signal data are strictly aligned in time.
[0041] The environmental state vector contains at least one or more of the following information: the ambient temperature value obtained by the in-vehicle temperature sensor; the air conditioning fan speed or setting value read by the air conditioning controller; the set temperature value of the vehicle's air conditioning system; and the air vent direction mode indicator (such as blowing towards the face, feet, defrosting, etc.). This environmental information can effectively characterize the environmental conditions inside the vehicle, especially the potential interference of the air conditioning system on the steering wheel area, providing crucial contextual information for subsequent multimodal fusion models.
[0042] In step S102 above, time-frequency transformation is performed based on the capacitor frequency signal to obtain a time-frequency spectrum.
[0043] Specifically, the capacitor frequency signal acquired in step S1011 can be subjected to time-frequency transformation to obtain a time-frequency spectrum.
[0044] The time-frequency transformation can be implemented using a Short-Time Fourier Transform (STFT) to convert the one-dimensional capacitor frequency signal into a two-dimensional time-frequency spectrum. Specifically, performing a Short-Time Fourier Transform on each frame of the capacitor frequency signal yields the spectral distribution corresponding to that frame. Arranging the spectral distributions of all frames in chronological order creates a two-dimensional matrix-like time-frequency spectrum S(t, f), where the horizontal axis represents time frames, the vertical axis represents frequency points, and each element in the time-frequency spectrum represents the signal energy intensity at the corresponding time frame and frequency point.
[0045] The time-spectrum graph can intuitively show the energy distribution characteristics of the capacitor frequency signal in different time and frequency dimensions. Different events such as hand contact, hand movement, and air conditioning wind impact will show different energy distribution patterns on the time-spectrum graph.
[0046] In step S103 above, the time spectrum and the environmental state vector are input into a preset deep spectrum fusion model to obtain the first off-hand detection result and the corresponding first confidence level.
[0047] Specifically, the time-spectrum graph generated in step S102 and the environmental state vector collected in step S1012 can be input together into a pre-trained deep spectrum fusion model to obtain the output first hand-drop detection result and the corresponding first confidence level. The first hand-drop detection result can be either "holding" or "dropped," and the first confidence level is a probability value between 0 and 1, representing the degree of certainty of the judgment result by the pre-trained deep spectrum fusion model.
[0048] The model structure and training steps of the preset deep spectrum fusion model will be detailed later, but will be briefly described here: The preset deep spectrum fusion model can adopt a dual-branch network structure design. One branch is used to receive the time spectrum as input to extract the frequency domain features in the time spectrum; the other branch is used to receive the environmental state vector as input to extract the environmental context features. Then, the deep features extracted by the two branches are fused. The fused features are then subjected to nonlinear transformation through several hidden layers, and finally the output layer outputs a binary classification probability value, namely the first off-hand detection result and its corresponding first confidence.
[0049] In step S104 above, timing features are extracted based on the capacitor frequency signal. Specifically, this includes the following steps S1041-S1042: S1041: Perform feature calculations on the capacitor frequency signal to obtain the moving average sequence, standard deviation sequence, and gradient sequence of the capacitor frequency signal.
[0050] Specifically, feature calculation can be performed within the capacitor frequency signal at a preset extraction frequency (e.g., 50 Hz), i.e., feature calculation is performed once every 0.02 seconds. For each feature calculation moment, a capacitor frequency signal segment of the preset feature calculation window length (e.g., 0.02 seconds to 0.1 seconds) is taken at that moment and before. The arithmetic mean of all values within the capacitor frequency signal segment is calculated as the moving average at that moment. The standard deviation of all values within the capacitor frequency signal segment is calculated as the standard deviation at that moment. The rate of change at that moment relative to the previous moment is calculated as the gradient, which can be obtained by dividing the difference between the current moment and the previous moment by the time interval.
[0051] Arrange the moving averages obtained from all feature calculation times in chronological order to obtain the moving average sequence; similarly, arrange all standard deviations in chronological order to obtain the standard deviation sequence; and arrange all gradient values in chronological order to obtain the gradient sequence.
[0052] S1042: Concatenate the moving average sequence, standard deviation sequence, and gradient sequence to obtain the time series features.
[0053] Specifically, after obtaining the moving average sequence, standard deviation sequence, and gradient sequence, the three sequences can be concatenated and fused along the feature dimension. For each feature calculation time, the corresponding moving average, standard deviation, and gradient value are combined into a three-dimensional feature vector.
[0054] Assuming a preset time window of 2 seconds and a preset extraction frequency of 50 Hz, the lengths of the moving average sequence, standard deviation sequence, and gradient sequence are all 100 (i.e., 2 seconds × 50 Hz). Concatenating the moving average sequence, standard deviation sequence, and gradient sequence column-wise yields a two-dimensional matrix of shape (100, 3), where 100 represents the time step and 3 represents the feature dimension. This matrix represents the time-series feature. This time-series feature fully preserves the dynamic changes of the capacitor frequency signal over time, as well as the statistical characteristics and trends of the signal at each time point, providing a standardized data format for subsequent input into the preset time-series feature model.
[0055] In step S105 above, the temporal features are input into a preset temporal feature model to obtain the second release detection result and the corresponding second confidence level.
[0056] Specifically, the temporal features obtained in step S1042 can be input into a pre-trained preset temporal feature model to obtain the output second hand release detection result and the corresponding second confidence level. The second hand release detection result can be either "holding" or "released", and the second confidence level is a probability value between 0 and 1, representing the degree of certainty of the preset temporal feature model's judgment result.
[0057] The model structure and training steps of the preset temporal feature model will be detailed later; here, only a brief description is provided: The preset temporal feature model can adopt a recurrent neural network structure, preferably a Long Short-Term Memory (LSTM) network or a Gated Recurrent Unit (GRU) network, to fully capture the temporal dependencies in the temporal features. The temporal features are input into the preset temporal feature model in the form of a two-dimensional matrix with shape (time steps, feature dimension), where the feature dimension is 3 (corresponding to the moving average, standard deviation, and gradient). The preset temporal feature model processes the input sequence step by step through recurrent layers to extract the dynamic change patterns in the temporal features; the output of the recurrent layer undergoes nonlinear transformation through several fully connected layers, and finally outputs a binary classification probability value, namely the second off-hand detection result and its corresponding second confidence score, by the output layer.
[0058] In step S106 above, the steering wheel hands-off detection result is obtained based on the first hands-off detection result, the first confidence level, the second hands-off detection result, and the second confidence level. In this embodiment of the invention, step S106 can be implemented in several different ways, including but not limited to three: The first way is confidence level comparison arbitration, specifically including steps S10601-S10602 below. The second way is weighted fusion arbitration, specifically including steps S10611-S10615 below. The third way is state machine arbitration, specifically including steps S10621-S10622 below. The first method: Confidence level comparison arbitration S10601: If the first confidence level is greater than the second confidence level, then the first hand-off detection result shall be taken as the steering wheel hand-off detection result.
[0059] Specifically, this can be achieved by comparing the first confidence level output by the preset deep spectrum fusion model with the second confidence level output by the preset temporal feature model. If the first confidence level is greater than the second confidence level, it indicates that at the current moment, the preset deep spectrum fusion model has a higher degree of certainty regarding its judgment result than the preset temporal feature model. In this case, the first hand-off detection result output by the preset deep spectrum fusion model is selected as the final steering wheel hand-off detection result. For example, if the first confidence level is 0.92 and the second confidence level is 0.78, meaning the first confidence level is greater than the second confidence level, then the first hand-off detection result is used as the final output.
[0060] S10601: If the second confidence level is greater than the first confidence level, then the second hand-off detection result shall be taken as the steering wheel hand-off detection result.
[0061] Specifically, if the second confidence level is greater than the first confidence level, it indicates that at the current moment, the degree of certainty of the judgment result by the preset temporal feature model is higher than that of the preset deep spectrum fusion model. In this case, the second hand-off detection result output by the preset temporal feature model is selected as the final steering wheel hand-off detection result. For example, if the first confidence level is 0.68 and the second confidence level is 0.89, that is, the second confidence level is greater than the first confidence level, then the second hand-off detection result is taken as the final output.
[0062] The second method: Weighted fusion arbitration S10611: Based on the historical accuracy of the preset deep spectrum fusion model and the preset time-series feature model under the corresponding working conditions of the environmental state vector, determine the first weight corresponding to the preset deep spectrum fusion model and the second weight corresponding to the preset time-series feature model.
[0063] Specifically, this can be achieved by determining the current operating condition category based on the environmental state vector, such as high temperature and high air volume operating condition, low temperature and calm air operating condition, and sudden temperature change operating condition. The historical detection accuracy rates of the preset deep spectrum fusion model and preset temporal feature model under various operating conditions are pre-calculated, and a mapping table between different operating conditions and model accuracy rates is established.
[0064] During real-time detection, the mapping table is queried based on the environmental state vector to obtain the historical accuracy of the preset deep spectrum fusion model under the corresponding environmental state vector conditions, which serves as the first weight. The historical accuracy of the preset temporal feature model under the corresponding environmental state vector conditions is used as the second weight. These two weights reflect the reliability of the detection results of the two models under the current environmental state vector conditions.
[0065] S10612: Based on the first weight and the first confidence level, obtain the first weighted confidence level.
[0066] Specifically, the first weight can be multiplied by the first confidence level to obtain the first weighted confidence level. The first weighted confidence level comprehensively considers the historical performance of the preset deep spectrum fusion model under the current environmental state vector and the first confidence level, and can more accurately reflect the reliability of the output results of the preset deep spectrum fusion model.
[0067] S10613: Based on the second weight and the second confidence level, the second weighted confidence level is obtained.
[0068] Specifically, the second weight can be multiplied by the second confidence level to obtain the second weighted confidence level. Similar to the first weighted confidence level, the second weighted confidence level comprehensively considers the historical performance of the preset time-series feature model under the current environmental state vector and the second confidence level.
[0069] S10614: If the first weighted confidence level is greater than the second weighted confidence level, then the first hand-off detection result shall be taken as the steering wheel hand-off detection result.
[0070] Specifically, the calculated first weighted confidence level can be compared with the second weighted confidence level. If the first weighted confidence level is greater than the second weighted confidence level, it indicates that the output of the preset deep spectrum fusion model is more reliable after comprehensively considering historical performance and confidence level. In this case, the first hands-off detection result output by the preset deep spectrum fusion model is selected as the final steering wheel hands-off detection result.
[0071] S10615: If the second weighted confidence level is greater than the first weighted confidence level, then the second hand-off detection result shall be taken as the steering wheel hand-off detection result.
[0072] Specifically, if the second weighted confidence level is greater than the first weighted confidence level, it indicates that the output result of the preset time series feature model is more reliable after comprehensively considering historical performance and confidence level. In this case, the second hands-off detection result output by the preset time series feature model is selected as the final steering wheel hands-off detection result.
[0073] In this embodiment, the second method of weighted fusion arbitration introduces dynamic weights based on historical accuracy under different operating conditions, enabling adaptive adjustment of the contributions of the two models. Compared to fixed weight allocation or simple confidence comparison, this method can dynamically adjust the fusion weights of the two models according to the current environmental conditions, making the detection results more dependent on the model that performs better under the current environment, thereby improving the adaptability and robustness of the detection method under different operating conditions. Simultaneously, the weighted fusion process integrates the historical performance and real-time confidence of the models, allowing for correction through historical weights when a single model experiences abnormal confidence fluctuations due to sudden interference. This avoids misjudgments caused by single confidence anomalies, further improving the accuracy and reliability of the steering wheel off-hand detection results.
[0074] The third method: state machine arbitration S10621: If the first hand-off detection result and the second hand-off detection result are the same, then the first hand-off detection result or the second hand-off detection result shall be taken as the steering wheel hand-off detection result.
[0075] Specifically, when the first hand-off detection result output by the preset deep spectrum fusion model is consistent with the second hand-off detection result output by the preset temporal feature model, it indicates that the two models have reached a consensus. In this case, there is no need for complex arbitration logic, and either the first or second hand-off detection result can be directly output as the final steering wheel hand-off detection result. For example, if both the first and second hand-off detection results are in the "hands-on" state, the final steering wheel hand-off detection result is "hands-on"; if both the first and second hand-off detection results are in the "hands-off" state, the final result is "hands-off". This method can quickly output the detection result when the model results are consistent, ensuring the system's response speed.
[0076] S10622: If the first and second steering wheel release detection results differ, the steering wheel release detection result is determined based on the historical steering wheel release detection results and the environmental state vector. This specifically includes the following steps S106221-S106222: S106221: If the historical steering wheel hands-off detection results remain stable within a preset time period, and the environmental state vector indicates that the vehicle is not in a preset high-interference condition, then the historical steering wheel hands-off detection results will be used as the steering wheel hands-off detection results.
[0077] Specifically, when the first and second hands-free detection results differ, a sequence of historical steering wheel hands-free detection results within a preset time period (e.g., 5 seconds) prior to the current moment can be obtained first. If this historical detection result sequence remains stable within the preset time period, i.e., consistently showing "hands on" or consistently showing "hands off," it indicates that the vehicle's steering wheel status was relatively stable over the previous period.
[0078] Simultaneously, the system combines the environmental state vector to determine whether the system is under a preset high-interference condition. Preset high-interference conditions may include air conditioning fan speed exceeding a preset threshold or air vents blowing directly onto the steering wheel. If the historical state is stable and the system is not currently under a preset high-interference condition, it tends to assume that the steering wheel hands-off detection result should not change abruptly. Therefore, the historical steering wheel hands-off detection result is used as the final steering wheel hands-off detection result output to avoid state jumps caused by instantaneous interference or occasional model errors.
[0079] S106222: If the environmental state vector represents that the vehicle is in a preset high-interference condition, then the steering wheel hand-off detection result is determined from the first and second hand-off detection results based on the first and second confidence levels.
[0080] Specifically, when the environmental state vector determines that the current operating condition is under a preset high interference condition (such as strong air conditioning blowing directly onto the steering wheel), it indicates that the capacitor frequency signal may be subject to strong environmental interference. At this time, the outputs of the preset deep spectrum fusion model and the preset time series feature model may be affected.
[0081] In this case, we can return to the confidence comparison arbitration logic described in the first method above, compare the first confidence level with the second confidence level, and select the hand-off detection result corresponding to the model with the higher confidence level as the final steering wheel hand-off detection result.
[0082] In this embodiment, the third state machine arbitration method constructs a decision-making mechanism with memory capabilities by introducing historical detection result sequences and environmental state information. When the outputs of the preset deep spectrum fusion model and the preset temporal feature model are consistent, the result is output directly, ensuring the system's response speed. When detection results conflict, a comprehensive judgment is made by combining the stability of historical states and the degree of current environmental interference. When the historical state is stable and there is no strong interference, the historical state is maintained, avoiding frequent state jumps caused by instantaneous interference. In scenarios with strong interference, it reverts to a confidence comparison strategy, ensuring that reasonable decisions can still be made in complex environments. This method achieves temporal smoothness and scenario adaptability of detection results, significantly improving the system's robustness and user experience.
[0083] In this embodiment of the application, before performing the multimodal information fusion-based steering wheel hands-off detection method in steps S101-S106 above, step S100 is also included: obtaining a preset deep spectrum fusion model and a preset temporal feature model, specifically including the following steps S1001-S1003: S1001: Under various operating conditions, acquire multiple historical environmental state vectors and multiple historical capacitance frequency signals collected by the capacitance sensor of the vehicle steering wheel.
[0084] Specifically, this can involve deploying data collection vehicles in real road testing environments or laboratory simulation environments to collect data under various operating conditions.
[0085] The data collection conditions need to cover a variety of typical driving scenarios, including but not limited to: different ambient temperatures (such as -20℃ to 40℃ range), different air conditioning settings (such as air conditioning off, low air volume, high air volume, different air outlet modes), different hand contact states (such as holding, releasing, lightly touching with fingers, holding in multiple positions with the palm), and different driving behaviors (such as driving straight, turning, bumpy roads), etc.
[0086] For each operating condition, the following data are collected simultaneously: raw capacitance frequency signals are acquired via the vehicle steering wheel capacitance sensor at a preset extraction frequency; environmental state vectors, including ambient temperature, air conditioning fan speed, air conditioning set temperature, and air vent direction, are acquired via the vehicle CAN bus; and real hand contact state labels are recorded at each moment through manual annotation or video monitoring, with the label value being "hand gripping" or "hand released". All collected historical capacitance frequency signal sequences, historical environmental state vectors, and real labels constitute the original dataset for subsequent model training.
[0087] After acquiring the raw data, it is preprocessed. Specifically, for each historical capacitor frequency signal sequence, a sliding window is used to extract the signal with a preset time window (e.g., 2 seconds) and a preset sliding step size (e.g., 0.1 seconds) to obtain multiple historical capacitor frequency signals. Each historical capacitor frequency signal corresponds to a capacitor frequency signal sequence within a preset time window.
[0088] Simultaneously, for each historical capacitance frequency signal, the corresponding environmental state vector is extracted from the historical environmental state vector and normalized. Each historical capacitance frequency signal, its corresponding environmental state vector, and its corresponding actual hand contact state are combined into a historical sample. This preprocessing operation is repeated to generate a large number of historical samples based on the continuous historical capacitance frequency signal sequence, which are then used for subsequent model training.
[0089] S1002: The model is trained based on multiple historical environmental state vectors and multiple historical capacitor frequency signals to obtain a preset deep spectrum fusion model.
[0090] Specifically, this can be achieved by constructing and training a pre-defined deep spectrum fusion model according to the following steps: The first step is model input construction: From the historical samples generated in step S1001, the historical capacitance frequency signal is extracted from each historical sample. Similar to step S102 above, a time-frequency transformation is performed on the historical capacitance frequency signal to obtain the historical time-frequency spectrum. Simultaneously, the corresponding historical environment state vector is extracted from each historical sample. Each historical time-frequency spectrum and its corresponding historical environment state vector are used as input to a training sample. The true label of this training sample is the actual hand contact state recorded in the historical samples.
[0091] The second step is model structure design: the pre-defined deep spectrum fusion model adopts a dual-branch network structure. Specifically, this includes: The first branch is a convolutional neural network branch, used to receive historical time-domain spectrograms as input. This branch can include, in sequence: an input layer for receiving historical time-domain spectrograms; multiple convolutional layers, each followed by a batch normalization layer and a ReLU activation function, for extracting local frequency domain features; pooling layers for dimensionality reduction and preservation of key features; and a Flatten layer for converting the two-dimensional feature map into a one-dimensional feature vector.
[0092] The second branch is a fully connected network branch, used to receive historical environment state vectors as input. This branch can consist of: an input layer for receiving the environment state vector; and multiple fully connected layers, each followed by a ReLU activation function to extract environmental context features.
[0093] The feature vectors output from the two branches are concatenated in the fusion layer to form a fused feature vector. The fused feature vector is then input into multiple fully connected hidden layers for nonlinear transformation. Finally, the output layer outputs a binary classification probability value through the Softmax activation function, which is the historical first slip-out detection result and the corresponding historical first confidence score.
[0094] The third step is model training. The constructed training samples are divided into training, validation, and test sets. The cross-entropy loss function is used as the optimization objective, and the Adam optimizer is used for parameter updates. Appropriate batch sizes and training epochs are set. During training, the loss value and accuracy on the validation set are monitored, and an early stopping strategy is employed to prevent overfitting. After training, the parameters of the model that performs best on the validation set are saved as the preset deep spectrum fusion model.
[0095] S1003: The model is trained based on multiple historical capacitor frequency signals to obtain a preset time-series feature model.
[0096] Specifically, this can be achieved by constructing and training a pre-defined time-series feature model according to the following steps: The first step is to construct temporal features: From the historical samples generated in step S1001, extract the historical capacitance frequency signal from each historical sample. Similar to step S104 above, perform temporal feature extraction on the historical capacitance frequency signal to obtain historical temporal features. Specifically, perform feature calculations within the historical capacitance frequency signal according to a preset extraction frequency. For each feature calculation time, take a segment of the historical capacitance frequency signal up to and including that time and the preset feature calculation window length. Calculate the arithmetic mean of this signal segment as the moving average, calculate the standard deviation of this signal segment, and calculate the rate of change of capacitance frequency at that time relative to the previous time as the gradient. Arrange the moving average, standard deviation, and gradient obtained from all feature calculation times in chronological order and concatenate them along the feature dimension to obtain a historical temporal feature matrix of shape (time step, 3). Use each historical temporal feature matrix as input to a training sample; the true label of this training sample is the actual hand contact state recorded in the historical samples.
[0097] The second step is model structure design: the pre-defined time-series feature model adopts a recurrent neural network structure. Specifically, this includes: The input layer is used to receive historical time-series features of shape (time steps, 3); The recurrent layer, using a long short-term memory network or gated recurrent units, sets the number of hidden units to capture the temporal dependencies and dynamic change patterns in the historical time series feature matrix. The recurrent layer can be set as a single layer or a multi-layer structure, and each recurrent network can be followed by a Dropout layer to prevent overfitting. Fully connected hidden layers are used to perform further nonlinear transformations on the deep features output by the recurrent layers. One or more fully connected layers can be set, each followed by a ReLU activation function. The output layer outputs binary classification probability values through the Softmax activation function, namely the historical second-to-last hand-drop detection result and the corresponding historical second-to-last confidence score.
[0098] The third step is model training. The constructed training samples are divided into training, validation, and test sets. The cross-entropy loss function is used as the optimization objective, and the Adam optimizer is used for parameter updates. Appropriate batch sizes and training epochs are set. During training, the loss value and accuracy on the validation set are monitored, and an early stopping strategy is employed to prevent overfitting. After training, the parameters of the model that performs best on the validation set are saved as the preset temporal feature model.
[0099] In this embodiment, the core of the method lies in performing time-frequency analysis on the original capacitance frequency signal, extracting its time-spectrum as frequency domain features, and simultaneously extracting the time-series statistical features of the capacitance frequency signal as time domain features. This constructs a dual-model parallel detection architecture based on a preset deep spectrum fusion model and a preset time-series feature model. The two models analyze from the frequency and time domains respectively, forming complementary advantages and redundant backups. Furthermore, an arbitration mechanism is introduced, using strategies such as confidence comparison, weighted fusion, or state machines to intelligently decide on the outputs of the two models, thereby improving detection accuracy while enhancing the system's fault tolerance and the reliability of the output results. The above technical solution achieves a leap from single scalar processing to multi-dimensional pattern analysis, fully utilizing the deep features of the capacitance signal and the vehicle's environmental context information.
[0100] Compared with existing technologies, this method exhibits significantly enhanced anti-interference capabilities. By directly learning from the spectrum and distinguishing between hand contact patterns and environmental interference patterns using a deep learning model, it effectively suppresses misjudgments caused by factors such as air conditioning vents and temperature changes, significantly improving detection accuracy and reliability. Simultaneously, based on a pre-defined deep spectrum fusion model, it perceives environmental state information, predicting interference before drastic changes in the capacitance signal, achieving proactive and adaptive compensation with a faster and more intelligent response. The deep learning model is suitable for characterizing complex nonlinear coupling relationships among multiple variables such as capacitance signals, temperature, and airflow, overcoming the inherent limitations of traditional linear compensation models and possessing powerful nonlinear processing capabilities.
[0101] Furthermore, this method reduces sensitivity to noise at single signal points by utilizing rich spectral and contextual information. After sufficient training, the pre-set deep spectral fusion model and the pre-set temporal feature model can adapt to various previously unseen complex operating conditions, exhibiting high robustness and strong generalization ability. The dual-model parallel architecture constitutes a redundant system; even if one model temporarily fails due to specific interference, the other model can still provide effective output. An arbitration mechanism further ensures that the system maintains high decision-making quality even in extreme cases, meeting the functional safety requirements for system reliability. Each model has its strengths: the spectral fusion model excels at distinguishing different modes of interference, while the temporal model excels at capturing the dynamic dependencies of signals over time. The combination of these two models gives this method excellent adaptability to various types of hand manipulation and environmental interference.
[0102] Example 2 Based on the same inventive concept, embodiments of the present invention also provide a steering wheel hands-off detection device based on multimodal information fusion, referring to... Figure 2 As shown, the device includes: The data acquisition module 101 is used to acquire an environmental state vector and a capacitance frequency signal of the vehicle steering wheel; wherein the capacitance frequency signal is used to characterize the contact state between the vehicle steering wheel and the driver's hand. The time-frequency conversion module 102 is used to perform time-frequency conversion based on the capacitor frequency signal to obtain a time-frequency spectrum. The first prediction module 103 is used to input the time spectrum map and the environmental state vector into a preset deep spectrum fusion model to obtain a first off-hand detection result and a corresponding first confidence level. Data extraction module 104 is used to extract time-series features based on the capacitor frequency signal; The second prediction module 105 is used to input the time series features into a preset time series feature model to obtain a second release detection result and a corresponding second confidence level. The result determination module 106 is used to obtain the steering wheel hand-off detection result based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level.
[0103] Example 3 Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the steering wheel hands-off detection method based on multimodal information fusion as described in Embodiment 1 above.
[0104] Example 4 Based on the same inventive concept, this embodiment of the invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steering wheel hands-off detection method based on multimodal information fusion as described in Embodiment 1 above.
[0105] Example 5 Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the steering wheel hands-off detection method based on multimodal information fusion as described in Embodiment 1 above.
[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0110] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A steering wheel hands-off detection method based on multimodal information fusion, characterized in that, The method includes: Acquire the environmental state vector and the capacitance frequency signal of the vehicle steering wheel; wherein the capacitance frequency signal is used to characterize the contact state between the vehicle steering wheel and the driver's hand; A time-frequency transformation is performed on the capacitor frequency signal to obtain a time-frequency spectrum. The time-spectrum diagram and the environmental state vector are input into a preset deep spectrum fusion model to obtain the first hand-drop detection result and the corresponding first confidence level. Timing features are extracted based on the capacitor frequency signal; The time-series features are input into a preset time-series feature model to obtain the second release detection result and the corresponding second confidence level. Based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level, the steering wheel hand-off detection result is obtained.
2. The method according to claim 1, characterized in that, The acquisition of the environmental state vector and the capacitance frequency signal of the vehicle steering wheel includes: Based on the capacitive sensor of the vehicle steering wheel, the capacitive frequency signal is collected with the current sampling time as the endpoint and a preset time window is traced back. Collect an environmental state vector that traces back a preset time window from the current sampling time as the endpoint; wherein, the environmental state vector includes at least one of ambient temperature, air conditioning air volume, air conditioning set temperature, and air outlet direction.
3. The method according to claim 1, characterized in that, The extraction of time-series features based on the capacitor frequency signal includes: The capacitor frequency signal is subjected to feature calculations to obtain the moving average sequence, standard deviation sequence, and gradient sequence of the capacitor frequency signal; The time series features are obtained by concatenating the moving average sequence, the standard deviation sequence, and the gradient sequence.
4. The method according to claim 1, characterized in that, The process of obtaining the steering wheel hand-off detection result based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level includes: If the first confidence level is greater than the second confidence level, then the first hand-off detection result is taken as the steering wheel hand-off detection result; If the second confidence level is greater than the first confidence level, then the second hand-off detection result is taken as the steering wheel hand-off detection result.
5. The method according to claim 1, characterized in that, The process of obtaining the steering wheel hand-off detection result based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level includes: Based on the historical accuracy of the preset deep spectrum fusion model and the preset temporal feature model under the working conditions corresponding to the environmental state vector, the first weight corresponding to the preset deep spectrum fusion model and the second weight corresponding to the preset temporal feature model are determined. Based on the first weight and the first confidence level, a first weighted confidence level is obtained; Based on the second weight and the second confidence level, a second weighted confidence level is obtained; If the first weighted confidence level is greater than the second weighted confidence level, then the first hands-off detection result is taken as the steering wheel hands-off detection result; If the second weighted confidence level is greater than the first weighted confidence level, then the second hands-off detection result is taken as the steering wheel hands-off detection result.
6. The method according to claim 1, characterized in that, The process of obtaining the steering wheel hand-off detection result based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level includes: If the first hand-off detection result and the second hand-off detection result are the same, then the first hand-off detection result or the second hand-off detection result shall be taken as the steering wheel hand-off detection result; If the first hand-off detection result and the second hand-off detection result are different, the steering wheel hand-off detection result is determined based on the historical steering wheel hand-off detection results and the environmental state vector.
7. The method according to claim 6, characterized in that, If the first hand-off detection result and the second hand-off detection result are different, the steering wheel hand-off detection result is determined based on historical steering wheel hand-off detection results and environmental information, including: If the historical steering wheel hands-off detection results remain stable within a preset time period, and the environmental state vector indicates that the vehicle is not in a preset high-interference condition, then the historical steering wheel hands-off detection results will be used as the steering wheel hands-off detection results. If the environmental state vector indicates that the vehicle is in a preset high-interference condition, then the steering wheel hands-off detection result is determined from the first hands-off detection result and the second hands-off detection result based on the first confidence level and the second confidence level.
8. The method according to claim 1, characterized in that, Before acquiring the environmental state vector and the capacitance frequency signal of the vehicle steering wheel, the method further includes: Under various operating conditions, acquire multiple historical environmental state vectors and multiple historical capacitance frequency signals collected by the capacitance sensor of the vehicle steering wheel; The preset deep spectrum fusion model is obtained by training the model based on the multiple historical environmental state vectors and the multiple historical capacitor frequency signals. The preset time-series feature model is obtained by training the model based on the multiple historical capacitor frequency signals.
9. A steering wheel hands-off detection device based on multimodal information fusion, characterized in that, include: The data acquisition module is used to acquire environmental state vectors and capacitance frequency signals of the vehicle steering wheel; wherein, the capacitance frequency signals are used to characterize the contact state between the vehicle steering wheel and the driver's hand. The time-frequency conversion module is used to perform time-frequency conversion based on the capacitor frequency signal to obtain a time-frequency spectrum. The first prediction module is used to input the time-spectrum map and the environmental state vector into a preset deep spectrum fusion model to obtain the first off-hand detection result and the corresponding first confidence level. The data extraction module is used to extract time-series features based on the capacitor frequency signal; The second prediction module is used to input the time series features into a preset time series feature model to obtain the second release detection result and the corresponding second confidence level. The result determination module is used to obtain the steering wheel hand-off detection result based on the first hand-off detection result, the first confidence level, the second hand-off detection result, and the second confidence level.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory. Its features are, The processor executes the computer program to implement the steering wheel hands-off detection method based on multimodal information fusion as described in any one of claims 1-8.