Intelligent cabin interaction method and system combined with digital twinning

Abstract

The invention discloses an intelligent cabin interaction method and system combined with digital twinning, and relates to the technical field of man-machine interaction, and the method comprises the steps: collecting the multi-dimensional perception data of a cabin, and constructing a layered cabin digital twinning model; performing scene analysis on the cockpit digital twin model, extracting driver state features, predicting driver potential interaction demands, generating a multi-modal interaction strategy set in combination with the driver potential interaction demands and a current driving scene, and performing simulation evaluation on the multi-modal interaction strategy set to obtain an interaction evaluation index set; performing multi-objective decision optimization under the driving safety constraint based on the interaction evaluation index set, and determining and adaptively executing an optimal interaction strategy; and acquiring actual execution data and driver feedback data of the optimal interaction strategy, and incrementally updating the hierarchical cabin digital twin model. According to the invention, the interaction adaptability and the safety reliability of the intelligent cabin system in a complex driving scene are obviously improved, and an effective technical support is provided for realizing intelligent man-machine cooperative driving.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to an intelligent cockpit interaction method and system combined with digital twins. Background Art

[0002] With the advancement of automotive intelligence, smart cockpit systems are constantly innovating and upgrading, gradually developing a variety of human-computer interaction technologies. Existing technologies achieve basic human-computer interaction capabilities by integrating multiple sensors to collect physiological and behavioral data from the driver, combined with multimodal interaction methods such as vision and voice. Digital twin technology, a new technological means of mapping the physical world into digital space, has been widely applied in industrial manufacturing, smart cities, and other fields. In the field of in-vehicle interaction, some research has begun to explore the use of digital twin technology to construct digital cockpit models for simulation and verification of interactive systems.

[0003] However, current smart cockpit interaction systems still face some common challenges: existing interaction systems often use pre-set, fixed strategies, which lag in responding to real-time changes in driving scenarios and the driver's personalized needs; the overall synergy of multimodal interaction methods needs to be improved; and system reliability and security need to be further strengthened. Furthermore, while digital twin technology has seen initial application in the automotive field, effectively integrating digital twin technology to enhance the interactive performance of smart cockpits remains a research direction worth exploring. Summary of the Invention

[0004] The present invention provides an intelligent cockpit interaction method and system combined with digital twins, which are used to solve the technical problems of insufficient adaptability of existing intelligent cockpit interaction systems and insufficient application depth of digital twin technology in the field of cockpit interaction.

[0005] In view of this, the first aspect of the present invention provides a smart cockpit interaction method combined with digital twins, comprising:

[0006] Collect multi-dimensional cockpit perception data and build a hierarchical cockpit digital twin model;

[0007] Conduct scenario analysis on the cockpit digital twin model to extract driver status characteristics and predict the driver's potential interaction needs;

[0008] Based on the cockpit digital twin model, combined with the driver's potential interaction needs and the current driving scenario, a multimodal interaction strategy set is generated. This multimodal interaction strategy set is simulated and evaluated to obtain an interaction evaluation index set.

[0009] Based on the interactive evaluation index set, multi-objective decision optimization is performed under driving safety constraints to determine and adaptively execute the optimal interactive strategy;

[0010] Collect actual execution data of the optimal interaction strategy and driver feedback data, and incrementally update the hierarchical cockpit digital twin model.

[0011] Optionally, building a hierarchical cockpit digital twin model includes:

[0012] Preprocess the collected cockpit multi-dimensional perception data to generate standardized data streams;

[0013] Build a hierarchical cockpit digital twin model architecture including driver layer, environment layer and vehicle layer;

[0014] Map the standardized data stream to the corresponding layers of the hierarchical cockpit digital twin model and establish the initial state of each layer;

[0015] Based on the state parameters of the environment layer and vehicle layer and the correlation between the layers, a driving scene feature extraction module is constructed for driving scene type identification and complexity assessment.

[0016] Optionally, predicting the driver's potential interaction needs includes:

[0017] Use the driving scenario feature extraction module in the hierarchical cockpit digital twin model to perform scenario analysis to obtain the driving scenario type, driving scenario complexity level, and environmental state characteristics;

[0018] Adjust the parameters of the preset contextual driver state cognition model based on the driving scenario type and environmental state characteristics;

[0019] Inputting the driver's physiological data and driver's behavioral data into the adjusted contextual driver state cognitive model to obtain the driver state characteristics;

[0020] Input the driver's state characteristics, driving scenario type and environmental state characteristics into the demand prediction model to obtain the probability value of potential interaction demand;

[0021] A priority threshold is set, and the probability values are filtered based on the priority threshold to obtain a list of potential interaction needs of the driver.

[0022] Optionally, the process of obtaining the interactive evaluation indicator set includes:

[0023] Retrieving a matching unimodal strategy from a preset interaction strategy library for each requirement in the driver's potential interaction requirement list;

[0024] Adjust the parameters of the retrieval single-modal strategy based on the driver's state characteristics, driving scene complexity, and vehicle operation status data;

[0025] Through multimodal collaborative optimization processing, the adjusted single-modal strategies are combined to form a multimodal fusion strategy tailored to various needs;

[0026] Perform cross-demand scheduling and conflict resolution for the multimodal fusion strategies corresponding to each demand, and build a multimodal interaction strategy set;

[0027] Input the multimodal interaction strategy set into the simulation environment of the hierarchical cockpit digital twin model to obtain the simulated driver's response data to each interaction strategy;

[0028] Based on the response data, the evaluation index of each interaction strategy in the multimodal interaction strategy set is calculated, and the evaluation index is standardized to generate a strategy-indicator correspondence set;

[0029] Perform single-strategy security analysis on each interaction strategy in the strategy-indicator correspondence set, identify risky interaction strategies, and feed back risky interaction strategies to the interaction strategy library for updating.

[0030] Optionally, determining and adaptively executing the optimal interaction strategy includes:

[0031] Determine the weight allocation scheme based on the complexity of the driving scenario, and dynamically assign weight coefficients to each evaluation indicator for driving scenarios of different complexities;

[0032] Set driving safety constraints and dynamically adjust them according to changes in driving scenarios;

[0033] The standardized evaluation indicators of each interaction strategy are weighted according to the weight distribution scheme to obtain the comprehensive score of each interaction strategy;

[0034] Based on driving safety constraints and comprehensive scores, a multi-strategy combination safety evaluation and screening and sorting are performed on each interactive strategy in the strategy-indicator correspondence set to obtain the candidate optimal strategy set;

[0035] Evaluate the matching degree between the hardware resource requirements of each interaction strategy in the candidate optimal strategy set and the available system resources, and select the interaction strategy with the highest matching degree as the optimal interaction strategy;

[0036] The optimal interaction strategy is converted into a modal execution instruction sequence and assigned to the corresponding execution unit according to the execution priority.

[0037] Optionally, incrementally updating the hierarchical cockpit digital twin model includes:

[0038] Preprocessing actual execution data and driver feedback data;

[0039] Compare the actual execution data with the corresponding evaluation indicator values in the interactive evaluation indicator set to calculate the execution deviation;

[0040] Adjust the parameters of the hierarchical cockpit digital twin model based on execution deviations;

[0041] Based on actual execution data and driver feedback data, optimize the inter-layer information transmission channels and influence mechanisms in the hierarchical cockpit digital twin model;

[0042] The updated model parameters are stored in the model library to complete the incremental update of the hierarchical cockpit digital twin model.

[0043] Optionally, the cockpit multi-dimensional perception data includes driver physiological data, driver behavior data, cockpit environment data, external environment data and vehicle operation status data.

[0044] A second aspect of the present invention provides an intelligent cockpit interaction system combined with digital twins, comprising:

[0045] A modeling module is used to collect multi-dimensional cockpit perception data and build a hierarchical cockpit digital twin model;

[0046] An analysis module is used to perform scenario analysis on the cockpit digital twin model, extract driver status characteristics, and predict the driver's potential interaction needs;

[0047] The evaluation module is used to generate a multimodal interaction strategy set based on the cockpit digital twin model, combined with the driver's potential interaction needs and the current driving scenario, and simulate and evaluate the multimodal interaction strategy set to obtain a set of interaction evaluation indicators;

[0048] The decision-making module is used to optimize multi-objective decisions under driving safety constraints based on a set of interactive evaluation indicators, determine and adaptively execute the optimal interactive strategy;

[0049] The update module is used to collect actual execution data of the optimal interaction strategy and driver feedback data, and incrementally update the hierarchical cockpit digital twin model.

[0050] The beneficial effects of the present invention are: the present invention realizes accurate mapping and real-time synchronization of the physical cockpit to the digital space by constructing a deep fusion architecture of a layered digital twin model and a multi-dimensional perception system. This architecture not only improves the system's perception and understanding capabilities of complex driving scenarios, but also enhances the dynamic representation of the driver-vehicle-environment interaction relationship through an inter-layer collaboration mechanism; innovatively proposes a scene perception-driven adaptive interaction mechanism, which dynamically adjusts the interaction strategy according to the scene complexity and driver status characteristics; designs an interaction decision-making framework based on multi-objective optimization, which achieves a balance between interaction efficiency and safety through dynamic weight allocation and real-time adjustment of safety constraints. This framework not only improves the system's ability to respond quickly to emergencies, but also enhances the execution effect of the interaction strategy through multimodal collaborative optimization; constructs a closed-loop model optimization mechanism, and continuously learns and updates based on actual interaction data, so that the system has self-optimization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 The figure is a flow chart of an intelligent cockpit interaction method combined with digital twins.

[0053] Figure 2 Construct a flow chart for a hierarchical cockpit digital twin model of an intelligent cockpit interaction method combined with digital twins.

[0054] Figure 3 The optimal interaction strategy determination and execution flow chart for an intelligent cockpit interaction method combined with digital twins. DETAILED DESCRIPTION

[0055] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] Example 1, with reference to Figures 1 to 3 , which is the first embodiment of the present invention, provides a smart cockpit interaction method combined with digital twins. The flow chart of the method is as follows Figure 1 As shown, the method includes, S1: collecting multi-dimensional perception data of the cockpit in real time and building a hierarchical cockpit digital twin model.

[0057] In one embodiment of the present invention, the flowchart for constructing a hierarchical cockpit digital twin model is as follows: Figure 1 As shown, the following steps are included:

[0058] S1.1: Collect cockpit multi-dimensional perception data in real time.

[0059] Among them, multi-dimensional cockpit perception data includes driver physiological data, driver behavior data, cockpit environment data, external environment data, and vehicle operating status data. Specifically, driver physiological data is collected through biosensors integrated into the cockpit, including but not limited to heart rate sensors and blood oxygen sensors; driver behavior data is collected through cockpit cameras, pressure sensor arrays, and on-board operation records; cockpit environment data is collected through temperature and humidity sensors, light sensors, air quality sensors, etc.; external environment data is collected through on-board cameras, radars, and other perception systems to collect road, traffic, weather, and other information; and vehicle operating status data is collected through the vehicle's OBD interface and various vehicle sensors.

[0060] In addition, the system is equipped with a data encryption transmission mechanism, multi-node redundant backup of key data, and an automatic switching mechanism for sensor failures.

[0061] Preferably, by building a comprehensive cockpit perception system that includes physiological, behavioral, environmental and vehicle status, combined with sensor redundancy and automatic fault switching mechanisms, the reliability of data collection is improved.

[0062] S1.2: Preprocess the collected cockpit multi-dimensional perception data to generate a standardized data stream.

[0063] Among them, preprocessing includes data denoising, outlier filtering, time synchronization processing and spatial registration.

[0064] S1.3: Build a hierarchical cockpit digital twin model architecture including the driver layer, environment layer, and vehicle layer.

[0065] First, determine the model architecture, including: building the basic framework of a hierarchical digital twin model, adopting a three-level structure of driver layer-environment layer-vehicle layer, in which the driver layer represents the driver's physiological and behavioral characteristics, the environment layer represents the cabin environment and external environment characteristics, and the vehicle layer represents the vehicle's operating status characteristics; establish a correspondence between the physical entities and digital representations of each layer, and use a data verification mechanism to ensure that the digital model can accurately reflect the status of the physical entities.

[0066] Secondly, construct the model structure of each layer, including: the driver layer digital model establishes a digital representation of the driver's physiological and behavioral state; the environment layer digital model establishes a digital representation of the cabin environment and external environment state; the vehicle layer digital model establishes a digital representation of the vehicle's operating status.

[0067] Thirdly, construct the inter-layer mapping relationship, including: establishing a two-way mapping mechanism between the driver layer and the environment layer, the environment layer and the vehicle layer, and the driver layer and the vehicle layer; designing a configurable inter-layer information transmission channel, which adopts a combination of event-driven and timed synchronization. The event trigger threshold can be dynamically adjusted, and the timed synchronization period can be configured. Data synchronization is performed when the state change trigger threshold or the timed synchronization period is reached, and data encryption transmission is implemented during the information transmission process.

[0068] Finally, the model operation mechanism is established, including: constructing a model state update method to achieve dynamic adjustment of state parameters, establishing a data storage structure with version identification to support the recording and traceability of model state, and setting model verification and correction rules to ensure model accuracy. Model update trigger conditions include: state deviation exceeding a preset threshold, the scheduled update period reaching its limit, and the occurrence of a key event.

[0069] It should be noted that the specific parameters of each layer of the model can be configured based on the application scenario to adapt to different cockpit interaction requirements.

[0070] S1.4: Map the standardized data stream to the corresponding layers of the hierarchical cockpit digital twin model and establish the initial state of each layer.

[0071] Specifically, according to the model architecture constructed in S1.3, the standardized data stream is time-series processed, including dividing the data stream into time windows, setting the overlapping intervals of adjacent windows, and time alignment; the driver's physiological data and behavioral data are mapped to the driver layer digital model, the physiological and behavioral characteristic parameters are extracted, and the initial state representation of the driver layer is established; the cabin environment data and external environment data are mapped to the environment layer digital model, the environment characteristic parameters are extracted, and the initial state representation of the environment layer is established; the vehicle operation status data is mapped to the vehicle layer digital model, the operation status characteristic parameters are extracted, and the initial state representation of the vehicle layer is established; finally, the correspondence between the initial state of each layer and the physical entity is checked, the consistency of the state mapping between layers is confirmed, and the initial state parameters are adjusted and optimized.

[0072] It should be noted that the mapping process must follow the inter-layer mapping relationship established in S1.3 to ensure the correct correspondence between the data flow and the model architecture. The setting of the initial state parameters can be adjusted according to the actual application scenario.

[0073] S1.5: Based on the state parameters of the environment layer and vehicle layer and the correlation between the layers, a driving scene feature extraction module is constructed for driving scene type identification and complexity assessment.

[0074] Specifically, a driving scene feature space is established, and cabin environment features (temperature, humidity, lighting, etc.) and external environment features (road type, traffic conditions, weather conditions, etc.) are extracted from the environment layer. Vehicle operation status features (vehicle speed, acceleration, steering angle, etc.) are extracted from the vehicle layer to form a scene feature vector; scene type recognition is performed based on the scene feature vector, scene matching is completed based on a preset scene type feature library, and the current driving scene type is output; based on the recognized driving scene type, the driving scene complexity score is calculated in combination with the scene feature vector to realize the evaluation of the complexity of the current driving scene.

[0075] Among them, the scene feature library adopts a hierarchical and progressive construction method: first, a basic scene library is established based on expert knowledge; second, the scene library is expanded and optimized through data collected from real vehicles; finally, machine learning methods are introduced to perform cluster analysis on scene features to form a dynamically updated scene feature library.

[0076] In addition, the driving scenario complexity score is calculated based on the environmental complexity (including cabin environment and external environmental factors), vehicle operating state complexity (including vehicle speed, acceleration, etc.), and driving task complexity (including steering, lane changing, braking, etc.). A weighted summation method is used to calculate the overall complexity score. Based on the score, the driving scenarios are divided into high level (score greater than 0.7), medium level (score between 0.3 and 0.7), and low level (score less than 0.3). In subsequent steps, the corresponding complexity level is determined based on this classification.

[0077] S1.6: Configure an update mechanism to keep the hierarchical cockpit digital twin model synchronized with the actual physical cockpit status.

[0078] Specifically, the update mechanism includes real-time comparison of the physical cockpit and digital model states; setting difference thresholds and update cycles for the driver, environment, and vehicle layers; triggering updates when the state difference exceeds the threshold or reaches the update cycle; and employing a layered, progressive approach to state synchronization to ensure the model accurately reflects the physical cockpit state. The update process also incorporates exception handling mechanisms, including retry attempts, data rollback, and warning prompts for failed updates.

[0079] Optimally, a three-level structural design based on the driver layer, the environment layer, and the vehicle layer achieves a systematic mapping from physical entities to digital space. Bidirectional mapping between layers and configurable information transmission channels enhance the model's flexibility and scalability. An update strategy combining event-driven and timed synchronization, coupled with a versioned data storage structure and a multi-level verification mechanism, effectively ensures real-time synchronization and state consistency between the digital model and the physical entity. On this basis, a multi-dimensional feature-based scene recognition mechanism and a hierarchical and progressive scene feature library were established. Combined with complexity assessment methods for environmental, vehicle, and task factors, this provides reliable data support for subsequent interaction strategy optimization.

[0080] S2: Perform scenario analysis on the cockpit digital twin model, extract driver status characteristics, and predict the driver's potential interaction needs.

[0081] In one embodiment of the present invention, step S2 includes the following steps:

[0082] S2.1: Use the driving scenario feature extraction module in the hierarchical cockpit digital twin model to perform scenario analysis to obtain the driving scenario type, driving scenario complexity level, and environmental state characteristics.

[0083] Among them, the driving scene type and driving scene complexity level are obtained based on the driving scene feature extraction module, and the environmental state features are directly obtained from the driving scene feature space, including cabin environment features and external environment features.

[0084] S2.2: Adjust the parameters of the preset contextualized driver state cognition model based on the driving scenario type and environmental state characteristics.

[0085] Specifically, based on the driving scenario type, a corresponding basic cognitive parameter template is selected from a pre-established scenario cognitive parameter library. The basic cognitive parameter template includes an attention allocation pattern and a prior probability of maneuver intention. The attention allocation pattern specifies the standard percentage of gaze time in key areas (the road ahead, the rearview mirror, and the instrument panel); the prior probability of maneuver intention reflects the probability distribution of various maneuvers (such as lane changing, deceleration, and steering) in the driving scenario type.

[0086] Furthermore, a weighted summation method is used to calculate the comprehensive influence coefficient based on environmental state characteristics, taking into account the influence of cabin temperature parameters, lighting parameters, and weather parameters. When environmental parameters exceed the preset range, their influence coefficients are increased accordingly. Based on the comprehensive influence coefficients, the basic cognitive parameter template is modified to generate contextualized driver state cognitive model parameters that adapt to the current driving scenario and environmental conditions. The contextualized driver state cognitive model parameters are then updated to the preset contextualized driver state cognitive model to adapt it to the current driving scenario environment. This technical solution, through a scenario-adaptive cognitive parameter template and a dynamic environmental impact assessment mechanism, enables the state recognition model to optimize parameters according to different driving scenarios and environmental conditions, thereby improving the model's environmental adaptability and state recognition accuracy.

[0087] S2.3: Input the driver's physiological data and driver's behavioral data into the adjusted contextual driver state cognition model to obtain the driver state characteristics.

[0088] Specifically, the pre-processed driver physiological and behavioral data are fed into an updated contextualized driver state cognitive model. Based on the eye movement data in the driver's physiological data, the actual percentage of gaze time in key areas is calculated and compared with the standard percentage of gaze time to obtain an attention distraction index. Based on a preset attention threshold, the driver's current attention level is determined, including three states: distracted, normal, and focused.

[0089] Furthermore, the system determines the driver's emotional state based on physiological data, including heart rate variability and facial expression analysis. Fatigue is determined using heart rate variability, while emotion recognition is performed based on facial expression trends. If the heart rate variability and facial expression results are consistent, the result is directly output. If they are inconsistent, a confidence-weighted vote is used to determine the final state.

[0090] Furthermore, by analyzing the driver's behavior data (steering wheel steering and pedal operation data) within a preset time window and combining it with the prior probability of operation intention, the driver's current operation intention is inferred.

[0091] Preferably, the state feature extraction scheme improves the reliability of driver state judgment through the collaborative analysis of multi-dimensional physiological indicators and behavioral data, combined with a confidence-weighted voting mechanism, and effectively reduces the risk of misjudgment caused by a single indicator.

[0092] S2.4: Input the driver state characteristics, driving scenario type, and environmental state characteristics into the demand prediction model to obtain the probability value of potential interaction demand.

[0093] Specifically, a multi-layer neural network model consisting of a feature processing layer, a feature fusion layer, and a demand prediction layer is constructed as a demand prediction model. The feature processing layer uses 128-dimensional hidden layer nodes for feature extraction. The feature fusion layer contains two fully connected network layers (256 dimensions and 128 dimensions) and uses the ReLU activation function. The demand prediction layer uses a 64-dimensional fully connected layer and a softmax function to output a four-dimensional probability vector. After encoding and standardizing the input features, independent feature extraction is performed through the feature processing layer, and nonlinear feature fusion is completed in the feature fusion layer. Finally, a four-dimensional probability vector is output in the demand prediction layer, representing the probability values of information acquisition demand, environmental adjustment demand, function control demand, and navigation interaction demand, respectively. The proposed multi-layer neural network model achieves a deep fusion of driver state features, scene features, and environmental features through hierarchical feature extraction and nonlinear feature fusion, thereby improving the accuracy of interactive demand prediction.

[0094] In addition, the demand prediction model is trained using historical annotated data, which includes samples of driver status-interaction demand correspondences under different scenario types, as well as interaction demand priority labels annotated by experts.

[0095] S2.5: Set a priority threshold, filter the probability values based on the priority threshold, and obtain a list of potential driver interaction needs.

[0096] The priority threshold is set based on the driving scenario complexity level and the driver's attention state. When the driving scenario complexity level is high or the driver's attention is distracted, the priority threshold is increased to reduce non-critical interaction needs. When the driving scenario complexity level is low and the driver's attention is focused, the priority threshold is lowered to provide more interaction options. When the driving scenario complexity level is medium or the driver's attention is normal, the baseline priority threshold is maintained. The needs in the driver's potential interaction need list are sorted from high to low according to their probability value. This dynamic priority adjustment mechanism adaptively sets thresholds based on the scenario complexity level and attention state, achieving intelligent screening of interaction needs while ensuring driving safety, effectively balancing the needs of interaction timeliness and driving safety.

[0097] S3: Based on the cockpit digital twin model, combined with the driver's potential interaction needs and the current driving scenario, a multimodal interaction strategy set is generated. The interaction safety simulation evaluation of the multimodal interaction strategy set is performed to obtain an interaction evaluation index set.

[0098] In one embodiment of the present invention, step S3 includes the following steps:

[0099] S3.1: For each requirement in the driver's potential interaction requirement list, a matching single-modal strategy is retrieved from the preset interaction strategy library. When a match fails, the degraded retrieval mode is automatically enabled and the basic safety strategy is returned.

[0100] Specifically, based on the driver's potential interaction needs list, a contextualized feature vector is constructed for each need. This feature vector includes four dimensions: need type, need priority, driver attention level, and driving scenario complexity. Need type and need priority are derived from the driver's potential interaction needs list in S2.5, driver attention level is based on the driver state feature extraction in S2.3, and driving scenario complexity is based on the scenario segmentation results in S1.5.

[0101] Furthermore, a demand-strategy matching score table is generated by calculating the cosine similarity between the contextualized feature vector and the applicable feature vectors of each unimodal strategy in the preset interaction strategy library. A dynamic matching threshold is set based on the complexity level of the driving scenario, with the matching threshold increasing with higher complexity levels. The demand-strategy matching scores are weighted, with the weight coefficient positively correlated with the priority of the demand. Unimodal strategies with a weighted matching score greater than or equal to the dynamic matching threshold are selected as the initial strategy set.

[0102] Furthermore, it is determined whether the preliminary strategy set is empty. When the preliminary strategy set is not empty, the preliminary strategy set is grouped by strategy type to form a visual strategy group, an auditory strategy group, and a tactile strategy group. The strategies in each strategy group are sorted from high to low according to the weighted matching score, and the top several strategies are extracted from each strategy group. The number of extracted strategies is determined according to the driver's current attention level, and the higher the attention level, the more strategies are extracted. When the preliminary strategy set is empty, the degraded retrieval mode is started, and the dynamic matching threshold is reduced by a preset ratio and the selection operation is re-executed. When the preliminary strategy set is still empty after degrading, the default strategy corresponding to the current demand type is extracted from the preset basic safety strategy library as an alternative strategy. Finally, the filtered strategies are integrated to form the final single-modal strategy set, and the weighted matching score, modal type, and corresponding driver demand item of each strategy are marked.

[0103] Preferably, the strategy retrieval scheme realizes the scenario-adaptive selection of interactive strategies through the construction of contextual feature vectors and dynamic matching threshold mechanism; at the same time, the designed degradation retrieval mechanism and basic security strategy guarantee ensure the reliability and security of interactive strategy selection.

[0104] S3.2: Adjust the parameters of the retrieved unimodal strategy based on the driver state characteristics, driving scenario complexity, and vehicle operation status data.

[0105] Specifically, parameter adjustments include the position and brightness of visual information, the volume and frequency of auditory information, and the intensity and duration of tactile information. For example, when adjusting visual information parameters, the lower the driver's attention level, the closer the display position to the central field of view is to enhance information perception. Visual information brightness is adjusted based on the complexity level of the driving scene, with higher complexity levels resulting in lower display brightness to reduce visual distraction. When adjusting auditory information parameters, the volume is adjusted based on the noise level in the vehicle's operating state data, with higher noise levels increasing the volume to ensure clear information transmission. The frequency of the prompt audio is adjusted based on the driver's emotional state, with the frequency decreasing when emotionally stressed to avoid exacerbating the driver's anxiety. When adjusting tactile information parameters, the intensity of tactile feedback is adjusted based on the complexity level of the driving scene, with higher complexity levels resulting in greater intensity to enhance warning effectiveness. The duration of tactile feedback is adjusted based on the driver's attention level, with lower attention levels resulting in longer duration to ensure effective information perception. The proposed adaptive parameter adjustment method dynamically optimizes the parameters of each modal information based on driver state, scene complexity, and vehicle operating status, improving the perception efficiency of interactive information and reducing driver distraction.

[0106] S3.3: Through multimodal collaborative optimization processing, the adjusted single-modal strategies are combined to form a multimodal fusion strategy tailored to various needs.

[0107] Specifically, the primary and secondary modalities are first determined based on the weighted matching scores of each modal strategy, and the modality with the highest score is used as the primary modality; secondly, the temporal relationship and complementary enhancement effect of the primary and secondary modalities are analyzed and optimized, such as the spatial consistency of visual cues and tactile feedback, and the semantic consistency of auditory cues and visual information.

[0108] S3.4: Perform cross-demand scheduling and conflict resolution for the multimodal fusion strategies corresponding to each demand, and build a multimodal interaction strategy set.

[0109] Specifically, the strategy execution priority is determined based on the demand type and timeliness; strategy execution conflicts in multi-demand scenarios are identified and handled, including timing conflicts (overlapping execution timings of multiple strategies), resource conflicts (multiple strategies competing for the same modal resources), and content conflicts (inconsistent information content of multiple strategies); corresponding scheduling rules are formulated to generate an execution schedule containing all demand response strategies, forming a coordinated and consistent set of multimodal interaction strategies.

[0110] Preferably, by determining the primary and secondary modalities based on matching scores and multimodal collaborative optimization, combined with cross-demand conflict identification and scheduling mechanisms, the collaborative effect of the multimodal interaction strategy is improved and modal conflicts in multi-demand scenarios are avoided.

[0111] S3.5: Input the multimodal interaction strategy set into the simulation environment of the hierarchical cockpit digital twin model to obtain the simulated driver's response data to each interaction strategy.

[0112] Response data includes reaction time, cognitive resource utilization, gaze deviation, and operational accuracy. Specifically, it captures the time interval from policy triggering to driver response, resource utilization at different cognitive stages, the driver's gaze deviation relative to a normal driving baseline, and the driver's accuracy in executing interactive commands.

[0113] S3.6: Based on the response data, calculate the evaluation indicators of each interaction strategy in the multimodal interaction strategy set, and generate a strategy-indicator correspondence set after standardizing the evaluation indicators.

[0114] The evaluation metrics include interaction timeliness, cognitive load, driver distraction, and interaction satisfaction prediction. Specifically, the comparison between reaction time and a safety threshold is converted into a timeliness score. A cognitive load metric is calculated by weighting the cognitive resources used in the perception, comprehension, and decision-making stages. Driver distraction is assessed based on the frequency, angle, and duration of gaze deviations. A multi-factor comprehensive evaluation model is used to predict interaction satisfaction. All metrics are normalized and mapped to the [0, 1] range.

[0115] S3.7: Perform single-strategy security analysis on each interaction strategy in the strategy-indicator correspondence set, identify risky interaction strategies, and feed the risky interaction strategies back to the interaction strategy library for updating.

[0116] The security analysis includes indicator threshold verification and combined risk assessment. Specifically, risky strategies are identified by comparing each evaluation indicator with its safety threshold. When multiple indicators simultaneously approach thresholds or a negative correlation trend appears between indicators, the corresponding strategy is marked as a potential risk strategy. Risky and potentially risky strategies are removed from the interaction strategy library or have their matching priority lowered. The designed security analysis mechanism, through indicator threshold verification and combined risk assessment, coupled with a dynamic update mechanism for the strategy library, effectively identifies and prevents potential interaction risks, thereby improving the security of the interaction system.

[0117] S4: Perform multi-objective decision optimization under driving safety constraints based on the interactive evaluation indicator set, determine and adaptively execute the optimal interactive strategy.

[0118] In one embodiment of the present invention, the optimal interaction strategy determination and execution flow chart is as follows: Figure 3 As shown, the following steps are included:

[0119] S4.1: Extract the standardized evaluation indicators corresponding to each interaction strategy from the strategy-indicator correspondence set.

[0120] S4.2: Determine the weight allocation scheme based on the complexity of the driving scenario, and dynamically assign weight coefficients to each evaluation indicator for driving scenarios of different complexities.

[0121] Specifically, based on the complexity level of the driving scenario, the weight of each indicator is determined through piecewise function mapping: when the complexity level of the driving scenario is high, the weight of safety-related indicators (cognitive load indicator and driving distraction indicator) increases accordingly, and the weight of interaction timeliness indicator and interaction satisfaction prediction indicator decreases accordingly; when the complexity level of the driving scenario is medium, the weight of each indicator is relatively evenly distributed; when the complexity level of the driving scenario is low, the weight of interaction timeliness indicator and interaction satisfaction prediction indicator increases accordingly, and the weight of safety-related indicators decreases accordingly; the output is an indicator weight vector that matches the complexity of the current driving scenario.

[0122] S4.3: Perform weighted calculation on the standardized evaluation indicators of each interaction strategy according to the weight distribution scheme determined in S4.2 to obtain the comprehensive score of each interaction strategy.

[0123] S4.4: Set driving safety constraints and dynamically adjust them according to changes in driving scenarios.

[0124] Driving safety constraints include a driver distraction threshold and a cognitive load threshold. The driver distraction threshold is determined by a combination of road type, traffic conditions, and weather conditions, while the cognitive load threshold is determined by the complexity of the driving scenario and the number of tasks currently being performed, and is negatively correlated with both.

[0125] Specific dynamic adjustment rules include: lowering the driving distraction index threshold when the road type is a highway and the traffic conditions are congested; lowering both the driving distraction index threshold and the cognitive load index threshold when the weather conditions are severe; raising the driving distraction index threshold when the road type is an urban road, the traffic conditions are sparse, and the weather conditions are good; lowering both the driving distraction index threshold and the cognitive load index threshold when the driver's attention level decreases, and maintaining both the driving distraction index threshold and the cognitive load index threshold when the driver's attention level is stable.

[0126] Preferably, the solution achieves a dynamic balance between interactive safety and interactive experience in different driving scenarios through a dynamic weight allocation mechanism driven by the scene complexity level, combined with a multi-factor driven safety constraint threshold adaptive adjustment strategy.

[0127] S4.5: Based on driving safety constraints and comprehensive scores, perform multi-strategy combination safety evaluation and screening and sorting for each interactive strategy in the strategy-indicator correspondence set to obtain the candidate optimal strategy set.

[0128] The candidate optimal strategy set consists of the top K interaction strategies that meet driving safety constraints and are ranked by comprehensive score. The K value is determined based on both the complexity of the driving scenario and the driver's attention state. A higher complexity level results in a smaller K value, reducing the number of available strategies. As the driver's attention state shifts from dispersed to concentrated, the K value increases, providing more interaction options.

[0129] S4.6: Evaluate the degree of match between the hardware resource requirements of each interaction strategy in the candidate optimal strategy set and the available system resources, and select the interaction strategy with the highest degree of match as the optimal interaction strategy.

[0130] Hardware resource requirements include computing, storage, and bandwidth requirements, while available system resources include currently available computing, storage, and bandwidth resources. The overall matching degree is calculated by calculating the ratio of each resource requirement to the corresponding available resources and performing a weighted calculation. The optimal interaction strategy is selected as the one with the highest overall matching degree and whose resource requirements do not exceed the available system resources. The designed resource matching evaluation mechanism comprehensively considers the usage of multiple types of system resources, ensuring the feasibility of interaction strategies within system resource constraints and improving system stability.

[0131] S4.7: Convert the optimal interaction strategy into a modal execution instruction sequence and assign it to the corresponding execution unit according to the execution priority.

[0132] Specifically, standard command information is generated based on the modality type and interaction parameters in the optimal interaction strategy. Combined with the established execution timing and priority rules, visual, auditory, and tactile command sequences are assigned to the corresponding control units for execution. Each execution unit executes the corresponding interaction action according to the timing and parameter configuration in the command sequence, continuously monitoring changes in the vehicle state during execution.

[0133] S4.8: Monitor vehicle state changes during strategy execution.

[0134] Specifically, when no emergency is detected, the execution unit continues to execute the interactive action according to the instruction sequence until completion; when an emergency is detected, the execution of the optimal interactive strategy is interrupted and switched to the safety-guaranteed interactive mode.

[0135] Emergency situations include collision warnings, lane departure warnings, emergency braking, and driver inattention, among other dangerous situations requiring immediate intervention. When the safety assurance interaction mode is triggered, normal interaction tasks, such as information acquisition and environmental adjustments, are immediately interrupted, while warning and reminder tasks related to vehicle safety are retained. Multimodal combined warning feedback allows for rapid driver awakening.

[0136] Ideally, the established interactive strategy execution and safety monitoring mechanism provides real-time safety assurance during the interactive process through continuous status monitoring and emergency response, improving the reliability and safety of the interactive system. This mechanism can promptly identify and respond to dangerous situations, ensuring driving safety is prioritized.

[0137] S5: Collect actual execution data of the optimal interaction strategy and driver feedback data, and incrementally update the hierarchical cockpit digital twin model.

[0138] In one embodiment of the present invention, step S5 includes the following steps:

[0139] S5.1: Collect actual execution data and driver feedback data of the optimal interaction strategy, and preprocess the actual execution data and driver feedback data.

[0140] Actual execution data includes interaction response time, interaction completion rate, system resource usage, and strategy execution status (normal completion / emergency interruption). Driver feedback includes operational responses and changes in physiological status. Preprocessing includes data denoising, outlier filtering, and time series alignment to generate a standardized execution performance evaluation data stream.

[0141] S5.2: Compare the actual execution data with the corresponding evaluation indicator values in the interactive evaluation indicator set and calculate the execution deviation.

[0142] Execution deviation is calculated based on the evaluation indicator system defined in S3.6, including the difference between the actual and predicted values for interaction timeliness, cognitive load, driving distraction, and interaction satisfaction prediction. For each indicator, execution deviation is calculated using a weighted mean square error, with the weight coefficient determined based on the indicator weight allocation scheme in S4.2. Execution data in emergency interruptions is treated as a special state and deviation is calculated and recorded separately.

[0143] S5.3: Adjust the parameters of the hierarchical cockpit digital twin model based on the execution deviation.

[0144] Specifically, execution deviations are used to optimize the layered cockpit digital twin model: physiological and behavioral state parameters at the driver level are adjusted to update the state recognition model; cockpit and external environment parameters at the environment level are optimized to refine feature classification standards; and operational state parameters at the vehicle level are updated to improve state quantification accuracy. The parameter adjustment process adheres to a data verification mechanism to ensure the accuracy of the correspondence between the digital model and the physical entity.

[0145] S5.4: Based on actual execution data and driver feedback data, optimize the inter-layer information transmission channels and influence mechanisms in the layered cockpit digital twin model.

[0146] Specifically, optimize inter-layer information transmission: adjust inter-layer bidirectional mapping parameters based on actual interaction data, and improve the transmission rules of state changes; optimize the triggering conditions of event-driven and timed synchronization mechanisms, and adjust the state synchronization cycle; improve exception handling strategies and improve the reliability of information transmission.

[0147] S5.5: Store the updated model parameters into the model library to complete the incremental update of the hierarchical cockpit digital twin model.

[0148] Specifically, the updated parameters are verified and deployed: the validity of the parameter update is verified through model verification rules; the parameter update history is recorded using version identifiers; the state is rolled back when the update triggers the preset threshold; security-related parameters are specially marked; and finally, the optimized model is synchronized to the operating environment.

[0149] Through these steps, we optimized the digital twin model based on actual interaction effects, improving the model's accuracy and adaptability to the actual interaction process. This incremental update mechanism ensures continuous improvement in model performance while ensuring the reliability and stability of the update process.

[0150] Furthermore, this embodiment also provides an intelligent cockpit interaction system combined with digital twins, including: a modeling module for collecting multi-dimensional perception data of the cockpit and constructing a hierarchical cockpit digital twin model; an analysis module for performing scenario analysis on the cockpit digital twin model, extracting driver status characteristics, and predicting the driver's potential interaction needs; an evaluation module for generating a multimodal interaction strategy set based on the cockpit digital twin model, combining the driver's potential interaction needs and the current driving scenario, and performing simulation evaluation on the multimodal interaction strategy set to obtain an interaction evaluation index set; a decision module for performing multi-objective decision optimization under driving safety constraints based on the interaction evaluation index set, determining and adaptively executing the optimal interaction strategy; an update module for collecting actual execution data of the optimal interaction strategy and driver feedback data, and incrementally updating the hierarchical cockpit digital twin model.

[0151] In summary, the present invention realizes the precise mapping and real-time synchronization of the physical cockpit to the digital space by constructing a deep fusion architecture of a layered digital twin model and a multi-dimensional perception system. This architecture not only improves the system's perception and understanding capabilities of complex driving scenarios, but also enhances the dynamic representation of the driver-vehicle-environment interaction relationship through an inter-layer collaboration mechanism. It innovatively proposes a scene perception-driven adaptive interaction mechanism, which dynamically adjusts the interaction strategy according to the scene complexity and driver status characteristics. It designs an interactive decision-making framework based on multi-objective optimization, which achieves a balance between interaction efficiency and safety through dynamic weight allocation and real-time adjustment of safety constraints. This framework not only improves the system's ability to respond quickly to emergencies, but also enhances the execution effect of the interaction strategy through multimodal collaborative optimization. It constructs a closed-loop model optimization mechanism, which continuously learns and updates based on actual interaction data, so that the system has self-optimization capabilities.

[0152] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A smart cockpit interaction method combined with digital twins, characterized in that: include: Collect multi-dimensional cockpit perception data and build a hierarchical cockpit digital twin model; Performing scenario analysis on the cockpit digital twin model to extract driver status characteristics and predict the driver's potential interaction needs; Based on the cockpit digital twin model, combined with the driver's potential interaction needs and the current driving scenario, a multimodal interaction strategy set is generated, and a simulation evaluation is performed on the multimodal interaction strategy set to obtain an interaction evaluation index set; Based on the interactive evaluation index set, multi-objective decision optimization is performed under driving safety constraints to determine and adaptively execute the optimal interactive strategy; The actual execution data of the optimal interaction strategy and the driver feedback data are collected, and the hierarchical cockpit digital twin model is incrementally updated.

2. The intelligent cockpit interaction method combined with digital twin according to claim 1 is characterized in that: The construction of the hierarchical cockpit digital twin model includes: Preprocess the collected cockpit multi-dimensional perception data to generate standardized data streams; Build a hierarchical cockpit digital twin model architecture including driver layer, environment layer and vehicle layer; Mapping the standardized data stream to the corresponding layer of the hierarchical cockpit digital twin model and establishing the initial state of each layer; Based on the state parameters of the environment layer and vehicle layer and the correlation between the layers, a driving scene feature extraction module is constructed for driving scene type identification and complexity assessment.

3. The intelligent cockpit interaction method combined with digital twin according to claim 1 is characterized in that: The predicted driver's potential interaction needs include: Utilizing the driving scene feature extraction module in the hierarchical cockpit digital twin model to perform scene analysis, obtaining the driving scene type, driving scene complexity level, and environmental state characteristics; Adjusting parameters of a preset contextualized driver state cognition model based on the driving scenario type and environmental state characteristics; Inputting the driver's physiological data and driver's behavioral data into the adjusted contextual driver state cognitive model to obtain the driver state characteristics; Inputting the driver state characteristics, the driving scenario type, and the environmental state characteristics into a demand prediction model to obtain a probability value of a potential interaction demand; A priority threshold is set, and the probability values are screened based on the priority threshold to obtain a list of potential driver interaction requirements.

4. The intelligent cockpit interaction method combined with digital twin according to claim 1 is characterized in that: The process of obtaining the interaction evaluation indicator set includes: Retrieving a matching unimodal strategy from a preset interaction strategy library for each requirement in the driver's potential interaction requirement list; Adjust the parameters of the retrieval single-modal strategy based on the driver's state characteristics, driving scene complexity, and vehicle operation status data; Through multimodal collaborative optimization processing, the adjusted single-modal strategies are combined to form a multimodal fusion strategy tailored to various needs; Perform cross-demand scheduling and conflict resolution for the multimodal fusion strategies corresponding to each demand, and build a multimodal interaction strategy set; Input the multimodal interaction strategy set into the simulation environment of the hierarchical cockpit digital twin model to obtain the simulated driver's response data to each interaction strategy; Based on the response data, the evaluation index of each interaction strategy in the multimodal interaction strategy set is calculated, and the evaluation index is standardized to generate a strategy-indicator correspondence set; Perform single-strategy security analysis on each interaction strategy in the strategy-indicator correspondence set, identify risky interaction strategies, and feed back risky interaction strategies to the interaction strategy library for updating.

5. The intelligent cockpit interaction method combined with digital twin according to claim 1 is characterized in that: Determining and adaptively executing the optimal interaction strategy includes: Determine the weight allocation scheme based on the complexity of the driving scenario, and dynamically assign weight coefficients to each evaluation indicator for driving scenarios of different complexities; Set driving safety constraints and dynamically adjust them according to changes in driving scenarios; The standardized evaluation indicators of each interaction strategy are weighted according to the weight distribution scheme to obtain a comprehensive score for each interaction strategy; Based on driving safety constraints and comprehensive scores, a multi-strategy combination safety evaluation and screening and sorting are performed on each interactive strategy in the strategy-indicator correspondence set to obtain the candidate optimal strategy set; Evaluate the matching degree between the hardware resource requirements of each interaction strategy in the candidate optimal strategy set and the available system resources, and select the interaction strategy with the highest matching degree as the optimal interaction strategy; The optimal interaction strategy is converted into a modal execution instruction sequence and assigned to the corresponding execution unit according to the execution priority.

6. The intelligent cockpit interaction method combined with digital twin according to claim 1 is characterized in that: Incrementally updating the hierarchical cockpit digital twin model includes: Preprocessing actual execution data and driver feedback data; Comparing the actual execution data with the corresponding evaluation index value in the interactive evaluation index set to calculate the execution deviation; Adjusting the parameters of the hierarchical cockpit digital twin model according to the execution deviation; Based on the actual execution data and driver feedback data, optimizing the inter-layer information transmission channels and influence mechanisms in the layered cockpit digital twin model; The updated model parameters are stored in the model library to complete the incremental update of the hierarchical cockpit digital twin model.

7. The intelligent cockpit interaction method combined with digital twin according to claim 1 is characterized in that: The cockpit multi-dimensional perception data includes driver physiological data, driver behavior data, cockpit environment data, external environment data and vehicle operation status data.

8. An intelligent cockpit interaction system combined with digital twins, characterized by: include: A modeling module is used to collect multi-dimensional cockpit perception data and build a hierarchical cockpit digital twin model; An analysis module, configured to perform scenario analysis on the cockpit digital twin model, extract driver status characteristics, and predict the driver's potential interaction needs; An evaluation module is configured to generate a multimodal interaction strategy set based on the cockpit digital twin model, in combination with the driver's potential interaction needs and the current driving scenario, and to perform simulation evaluation on the multimodal interaction strategy set to obtain an interaction evaluation index set; The decision-making module is used to optimize multi-objective decisions under driving safety constraints based on a set of interactive evaluation indicators, determine and adaptively execute the optimal interactive strategy; An update module is used to collect actual execution data of the optimal interaction strategy and driver feedback data, and incrementally update the hierarchical cockpit digital twin model.

Images