Vehicle lateral control method, controller and automobile

Abstract

The present application relates to a kind of vehicle square control control method, controller and car, comprising: when user triggers square control button, controller obtains multimodal user behavior data;Controller inputs multimodal user behavior data and the function information of square control button into the arbitration model of pre-set, obtains the target function and confidence of square control button triggered by user;Controller triggers the confirmation or execution of target function according to target function and confidence.The present application realizes the effect of improving square control button function determination efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicles, specifically to a vehicle steering control method, controller, and vehicle. Background Technology

[0002] With the increasing prevalence of intelligent vehicles, the functions integrated into the vehicle cabin are becoming increasingly complex. Steering wheel controls (hereinafter referred to as "steering wheel controls") no longer only perform simple tasks such as traditional volume adjustment and answering phone calls, but also integrate multiple functions including media control, navigation interaction, driver assistance activation, AI voice wake-up, and view switching. In this multi-functional integrated environment, the complexity of operation has increased significantly. However, existing steering wheel control interaction logic often uses an arbitration method of fixed button mapping and nested logic to handle user operations, resulting in high operational complexity. Summary of the Invention

[0003] One objective of this invention is to provide a vehicle steering control method to solve the problem of high operational complexity caused by the complex steering interaction logic in the prior art; another objective is to provide a controller; and a third objective is to provide a vehicle.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A vehicle lane control method, comprising:

[0006] When a user triggers a control button, multimodal user behavior data is acquired;

[0007] The multimodal user behavior data and the functional information of the control button are input into a preset arbitration model to obtain the target function and confidence level of the control button triggered by the user.

[0008] Based on the target function and the confidence level, the confirmation or execution of the target function is triggered.

[0009] Based on the above technical means, by acquiring multimodal user behavior data when the user triggers the control button, inputting it and the button function information into an arbitration model to obtain the target function and confidence level, and triggering function confirmation or execution accordingly, the method can achieve the effect of accurately identifying user intent and accurately triggering the corresponding function of the control button.

[0010] Furthermore, the functional information of the control button includes multiple functions corresponding to the control button; the multimodal user behavior data and the functional information of the control button are input into a preset arbitration model to obtain the target function and confidence level of the control button triggered by the user, including:

[0011] The arbitration model is used to process the multimodal user behavior data to obtain the user's behavior feature vector.

[0012] The behavioral feature vector and the multiple functions corresponding to the control button are input into the fully connected layer of the arbitration model to obtain the confidence level of each function;

[0013] The function with the highest confidence level is identified as the target function.

[0014] Based on the above technical means, by using an arbitration model to extract behavioral feature vectors from multimodal user behavior data, and then processing them through a fully connected layer to obtain the confidence scores of each function of the control button, and selecting the maximum value to determine the target function, the effect of accurately identifying the specific function corresponding to the user's intention to trigger the control button can be achieved.

[0015] Furthermore, using the arbitration model, the user behavior data for each modality is processed to obtain the user's behavior feature vector, including:

[0016] Using the arbitration model, the behavioral data of each modality in the user behavior data are normalized and encoded to obtain modal feature data for each modality;

[0017] The modal feature data of each modality are input into the fusion layer of the arbitration model to obtain the user's behavioral feature vector.

[0018] Based on the above technical means, by using an arbitration model to standardize and encode the modal data of each modality in user behavior data to obtain modal feature data, and then inputting it into the fusion layer to generate a behavior feature vector, the multimodal user behavior data can be efficiently transformed into a unified behavior feature vector to support subsequent accurate intent recognition.

[0019] Furthermore, the user behavior data for each modality includes driver head video data, vehicle control data, user historical behavior data, and voice data; using the arbitration model, the behavior data for each modality in the user behavior data is standardized and encoded to obtain modal feature data for each modality, including:

[0020] Based on the driver's head video data, the temporal feature data of the driver's head video data is extracted using the memory network module in the arbitration model;

[0021] The vehicle control data is input into the encoding module in the arbitration model, and the vehicle control data is converted into corresponding encoded feature data.

[0022] Based on the user's historical behavior data and voice data, the semantic feature data of the user's historical behavior data and voice data are extracted using the perceptual network module in the arbitration model.

[0023] Based on the above technical means, by using the memory network module in the arbitration model to extract the temporal features of the driver's head video data, the encoding module to convert the vehicle control data into encoded features, and the perception network module to extract the semantic features of the user's historical behavior data and voice data, the effect of comprehensively and accurately acquiring the modal feature data of multimodal user behavior data can be achieved.

[0024] Furthermore, based on the driver's head video data, the memory network module in the arbitration model is used to extract the temporal feature data of the driver's head video data, including:

[0025] Identify the pupil position at each moment in the driver's head video data to obtain an eye-tracking data sequence;

[0026] The facial expressions at each moment in the driver's head video data are identified to obtain an expression data sequence;

[0027] Identify the head posture at each moment in the driver's head video data to obtain a posture data sequence;

[0028] The memory network module in the arbitration model is used to process the eye-tracking data sequence, the facial expression data sequence, and the pose data sequence to obtain the temporal feature data.

[0029] Based on the aforementioned technical means, by identifying eye movements, facial expressions, and head postures from driver head video data and forming corresponding sequences, and then using the memory network module of the arbitration model to process these sequences to obtain temporal feature data, the effect of comprehensively and accurately extracting temporal features from driver head video data can be achieved to assist in subsequent behavioral intent analysis.

[0030] Furthermore, the modal feature data of each modality are input into the fusion layer of the arbitration model to obtain the user's behavioral feature vector, including:

[0031] A multi-head attention mechanism is adopted, and the modal feature data of each modality are aligned and fused based on the temporal information of each modality to obtain the first fused feature;

[0032] A cross-attention mechanism is used to align and fuse the semantic information of the modal feature data of each modality to obtain a second fused feature;

[0033] The first fusion feature and the second fusion feature are fused together to obtain the user's behavioral feature vector.

[0034] Based on the above technical means, the first fused feature is obtained by aligning and fusing the feature data of each modality based on time information using a multi-head attention mechanism, and the second fused feature is obtained by aligning and fusing the feature data based on semantic information using a cross-attention mechanism. The two are then fused to obtain the user behavior feature vector, thereby achieving the effect of fully integrating multimodal feature data and generating a comprehensive representation of user behavior feature vector.

[0035] Furthermore, based on the target function and the confidence level, triggering the confirmation or execution of the target function includes:

[0036] If the confidence level is greater than or equal to a preset threshold, the execution of the target function is triggered;

[0037] If the confidence level is less than the preset threshold, then the confirmation of the target function is triggered.

[0038] Based on the aforementioned technical means, by comparing the confidence level of the target function with the preset threshold, the execution or confirmation operation of the target function is triggered respectively, thereby achieving the effect of flexibly controlling the function triggering method according to the accuracy of intent recognition, so as to improve system security and user experience.

[0039] Furthermore, the method also includes:

[0040] If the confirmation result of the target function is cancellation, then obtain the first operation successfully performed by the user within the preset time period, and establish the association between the user behavior data and the operation;

[0041] The system periodically prompts users to view and confirm the correlation between the user behavior data and the operation; and uses the confirmed correlation between the user behavior data and the operation to optimize the arbitration model.

[0042] Based on the aforementioned technical means, by obtaining the user's first successful operation within a preset time period when the target function is confirmed to be canceled and establishing an association, periodically prompting the user to confirm the association relationship, and using the confirmed association data to optimize the arbitration model, the user behavior analysis model can be dynamically improved to enhance the accuracy of intent recognition.

[0043] A controller includes: a memory and a processor;

[0044] The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the aforementioned method.

[0045] An automobile includes a vehicle body, wherein the aforementioned controller is provided in the automobile.

[0046] The beneficial effects of this invention are:

[0047] This invention acquires driver behavior data through a DMS system and combines it with an AI model for intent recognition. It then intelligently judges and triggers functions based on click behavior on control buttons, thereby replacing and optimizing the original complex arbitration logic. This invention not only significantly simplifies the arbitration logic in system development but also improves the user experience, making vehicle function calls more natural and efficient, while reducing the probability of accidental touches and learning costs. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the vehicle steering control method provided by the present invention.

[0049] Figure 2 A schematic diagram of the arbitration model provided by this invention;

[0050] Figure 3 This is a schematic diagram of the user operation process provided by the present invention;

[0051] Figure 4 This is a schematic diagram of the vehicle steering control device provided by the present invention;

[0052] Figure 5 This is a schematic diagram of the controller provided by the present invention. Detailed Implementation

[0053] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0054] With the increasing prevalence of smart cars, the functions integrated into the vehicle cabin are becoming increasingly complex. Steering wheel controls (hereinafter referred to as "steering wheel controls") no longer only perform simple tasks such as traditional volume adjustment and answering phone calls, but also integrate multiple functions including media control, navigation interaction, driver assistance activation, AI voice wake-up, and view switching. In this multi-functional integrated environment, the complexity of operation has increased significantly.

[0055] However, existing steering wheel control interaction logic often uses a fixed button mapping and nested logic arbitration method to handle user operations. For example, a certain button may correspond to different functions in different application scenarios such as navigation mode, media mode, and call mode. The system needs to make judgments based on multiple conditions, such as the current vehicle system status, the foreground application stack, and the user's usage context, and then trigger different functions. This type of arbitration logic becomes increasingly complex as functions are added, which not only increases development costs and system complexity, but also creates obstacles for users. Especially for users unfamiliar with vehicle system systems, they may not be able to understand how certain functions are triggered by the steering wheel control.

[0056] Furthermore, users find it difficult to stare at the screen for extended periods while driving to switch menus or understand logic. Given the current system's inability to accurately interpret user intent, accidental touches or erroneous function triggers are frequent, severely impacting user experience and potentially posing safety hazards.

[0057] To address the aforementioned issues, this invention proposes a vehicle steering control method that combines a Driver Monitoring System (DMS) with AI technology. This method can identify the user's current intent in real time. Based on factors such as facial expressions, gaze direction, historical behavior, current driving state, and application scenario, it automatically infers the user's click purpose, thereby triggering the most appropriate function. This makes the steering control logic more intelligent, the user experience more user-friendly, and the arbitration logic simpler. The proposed steering control method can acquire driver behavior data through the DMS system and combine it with an AI model for intent recognition. It intelligently judges and triggers functions based on the click behavior of steering buttons, thus replacing and optimizing the original complex arbitration logic.

[0058] In this invention, the controller can use a Direct Motion Monitor (DMS) to collect real-time behavioral data such as the user's eye movement trajectory, facial expressions, head posture, blink frequency, and lip movements. The controller can also collect in-vehicle voice recognition data, the current system foreground status, and application usage frequency in real time, analyzing the data to obtain contextual information. The controller can utilize a deep neural network to construct an arbitration model for intent recognition. This arbitration model is used to automatically infer the potential purpose of the user's current click on the steering wheel control button based on multimodal data.

[0059] The use of this arbitration model allows the controller to no longer directly process user actions based on traditional pattern matching. Instead, it first determines the user's intent through the arbitration model and then selects the function that best matches that intent for triggering. This method can also retain a priority list of candidate functions for redundant switching or feedback confirmation.

[0060] Furthermore, this invention can also achieve an adaptive learning mechanism through online learning, gradually optimizing the accuracy of individual user intent prediction as users accumulate experience, thus establishing a personalized arbitration path. Additionally, when the confidence level of the intent is insufficient, it triggers a conservative mode that selects the function with the least interference or triggers voice confirmation. This conservative mode's safety redundancy mechanism ensures system stability and user safety.

[0061] In summary, the above-mentioned method not only significantly simplifies the arbitration logic in system development, but also improves the user's operating experience, making vehicle function calls more natural and efficient, while reducing the probability of accidental touches and learning costs.

[0062] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0063] Figure 1 This is a flowchart illustrating the vehicle steering control method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0064] S101. When the user triggers the control button, obtain multimodal user behavior data.

[0065] For example, the controller monitors the trigger signals of the steering wheel control buttons in real time through a hardware interface. When the controller detects that the steering wheel control button has been triggered, it can activate a multimodal sensor to collect behavioral feature data of the user during operation.

[0066] In one implementation, the steering wheel control buttons are physical buttons or touch-sensitive areas on the steering wheel. Users can quickly control preset functions by clicking these buttons. For example, they can adjust the volume, activate the voice assistant, and control cruise control.

[0067] In one implementation, the multimodal sensor comprises multiple sensors with various modes. For example, it may include a pressure sensor, a touch sensor, a camera, and a microphone.

[0068] In one implementation, the multimodal user behavior data consists of user behavior data collected by multiple sensors across various modalities. Examples include driver headshots, driver speech data, and voice data from inside the passenger cabin.

[0069] In one implementation, a pressure sensor and / or a capacitive touch sensor can be embedded inside the steering wheel control button. When a user touches or presses the steering wheel control button, the controller can detect a trigger signal.

[0070] In one implementation, the controller can also track hand gestures using a miniature camera built into the steering wheel to detect whether the steering wheel control buttons are triggered.

[0071] In one implementation, the controller can also detect minute vibrations of the steering wheel through an inertial measurement unit (IMU) to help determine whether the operation is accidental or intentional.

[0072] S102. Input the multimodal user behavior data and the function information of the control button into the preset arbitration model to obtain the target function and confidence level of the control button triggered by the user.

[0073] For example, the controller can input the multimodal data collected in step S101 above, along with the preset function list of the control buttons, into the arbitration model. This arbitration model can analyze user intent through feature extraction, pattern matching, and confidence calculation, and output the most likely target function and its corresponding confidence level.

[0074] In one implementation, the arbitration model is a pre-trained deep learning model used to analyze user intent.

[0075] In one implementation, the confidence level is the evaluation result of the arbitration model on the functional output indicated in the function information of the control button. This evaluation result is used to indicate the probability that the user intends to use the function. The higher the confidence level, the higher the probability that the user intends to use the function. The confidence level is a value between 0 and 1.

[0076] In one implementation, a control button can correspond to different functions in different scenario functional modes. These different functions can form a function list for the control button. This function list can include multiple functions. Specifically, in a scenario's functional mode, one control button typically corresponds to one function.

[0077] In one implementation, the target function of the control button is the function with the highest confidence among the functions of the control button.

[0078] S103. Based on the target function and confidence level, trigger the confirmation or execution of the target function.

[0079] For example, the controller first determines, based on the confidence level, whether it needs to confirm the target function or execute the target function. If it needs to execute the target function, the controller can directly trigger the execution of the target function. If it needs to confirm the target function, the controller can trigger the confirmation operation of the target function.

[0080] In one implementation, the controller can determine whether to trigger confirmation or execution of the target function based on a preset threshold.

[0081] In this example, by using an arbitration model to process multimodal user behavior data, the means to determine the target function and processing method are improved, thereby increasing the efficiency and accuracy of determining the function of the control button, enhancing the user experience, and reducing the complexity of confirming the function of the control button.

[0082] In one example, the function information of the control button includes multiple functions corresponding to the control button. In step S102 above, the process of the arbitration model processing multimodal user behavior data may include:

[0083] S1021. Using an arbitration model, process multimodal user behavior data to obtain user behavior feature vectors.

[0084] For example, the controller can input multimodal user behavior data into the arbitration model for feature extraction to obtain user behavior feature vectors.

[0085] In one implementation, the multimodal user behavior data input to the arbitration model can be data that conforms to the input format required by the arbitration model after preprocessing.

[0086] In one implementation, the arbitration model can perform feature extraction on each user behavior data and then use preset feature fusion parameters to fuse the features of the multimodal data to obtain the final behavior feature vector.

[0087] In one implementation, the arbitration model can directly use preset feature fusion parameters to fuse multimodal user behavior data to obtain the final behavior feature vector.

[0088] S1022. Input the behavioral feature vector and the multiple functions corresponding to the control buttons into the fully connected layer of the arbitration model to obtain the confidence level of each function.

[0089] For example, the controller can input the behavioral feature vector along with a list of functions included in the control buttons into the arbitration model. This arbitration model can then generate a confidence score for each function through its fully connected layers.

[0090] In one implementation, the multiple functions corresponding to the control button may include parameter information for those multiple functions.

[0091] In one implementation, the fully connected layer can calculate the matching score between each feature and the feature vector using linear transformations and nonlinear activation functions, ultimately outputting the confidence score of each feature. The fully connected layer is a layer in the network that connects all inputs and outputs, used to learn the global mapping relationship between features and features.

[0092] S1023. Determine the function with the highest confidence level as the target function.

[0093] For example, the controller sorts the functions of each control button according to their confidence level, and selects the function with the highest confidence level as the final target function.

[0094] In this example, by using fused multimodal user behavior data, the confidence level of each function of the control button is calculated, and the target function is determined based on the confidence level. This method achieves accurate analysis of the target function of the control button, improves the accuracy of target function recognition, and enhances user experience.

[0095] In one example, in step S1021 above, the controller uses an arbitration model to process the user behavior data of each modality to obtain the user's behavior feature vector. The specific process includes:

[0096] S10211. Using an arbitration model, the behavioral data of each modality in the user behavior data is standardized and encoded to obtain the modal feature data of each modality.

[0097] For example, the controller first performs modality separation on the collected multimodal user behavior data, and designs an independent standardized encoding process for the data characteristics of each modality. The controller can encode the user behavior data for each modality to obtain the modality feature data for each modality.

[0098] For example, pressure data can be normalized to scale it to the range of 0 to 1. Another example is performing time-series alignment and speed calculation on touch trajectories. Yet another example is extracting semantic feature vectors from voice commands after converting them into text using Automatic Speech Recognition (ASR).

[0099] In one implementation, the modal feature data for each modality can have a fixed-dimensional numerical representation. For example, pressure features are 16-dimensional vectors, and speech features are 64-dimensional vectors, etc.

[0100] In one implementation, the controller can achieve normalized coding by converting raw data of different dimensions, distributions, or formats into a uniform numerical range or structure, thereby eliminating the impact of intermodal differences on subsequent fusion.

[0101] For example, the controller can calculate the mean, variance, peak value, and rate of change of pressure values, and concatenate them into a feature vector to obtain the modal feature data of the pressure mode.

[0102] For example, the controller can extract temporal features such as the coordinate sequence of the touch point, movement speed, acceleration, and dwell time to obtain the modal feature data of the touch mode.

[0103] For example, the controller can use a pre-trained speech coding model to convert audio into semantic embedding vectors. These semantic embedding vectors can then serve as modal feature data for the speech modality.

[0104] S10212. Input the modal feature data of each modality into the fusion layer of the arbitration model to obtain the user's behavioral feature vector.

[0105] For example, the controller inputs the feature data from each modality into the fusion layer of the arbitration model. The fusion layer dynamically integrates the multimodal features through weighted summation, attention mechanisms, or deep neural networks to generate a comprehensive behavioral feature vector. This vector retains the key information of each modality while eliminating redundancy or conflicts between modalities.

[0106] In one implementation, the fusion layer is the core module in the arbitration model responsible for integrating multimodal features, generating a unified representation by learning the correlation between modalities.

[0107] In one implementation, the fusion layer can first directly concatenate the feature vectors of each modality, and then input them into a single-layer multilayer perceptron (MLP) for dimensionality reduction to obtain the final behavioral feature vector.

[0108] In another implementation, the fusion layer can input each modality feature into an independent sub-network, extract high-level features, and then concatenate them to obtain the final behavioral feature vector.

[0109] In another implementation, the fusion layer can introduce a self-attention mechanism to dynamically adjust the weights of each modality feature to obtain the final behavioral feature vector.

[0110] In another implementation, the fusion layer can construct multimodal features into a graph structure, and generate fused features after propagating information through a graph neural network (GNN) to obtain the final behavioral feature vector.

[0111] In this example, by using multimodal data normalization coding and dynamic fusion, we can achieve high-precision extraction of user behavior features and cross-modal information complementarity, thereby improving the robustness of the control function parsing.

[0112] In one example, user behavior data for each modality includes driver head video data, vehicle control data, user historical behavior data, and voice data.

[0113] In step S10211 above, the controller uses an arbitration model to normalize and encode the behavioral data of each modality in the user behavior data, obtaining modal feature data for each modality, specifically including:

[0114] S102111. Based on the driver's head video data, the memory network module in the arbitration model is used to extract the temporal feature data of the driver's head video data.

[0115] For example, the controller can acquire driver head video data via an in-vehicle camera. The controller can then input this driver head video data into the memory network module of the arbitration model. This memory network module analyzes the dynamic changes between video frames through a temporal modeling mechanism, extracting feature vectors with temporal continuity as temporal feature data.

[0116] In one implementation, the memory network module can be a sub-network in the arbitration model that specifically processes time-series data, capturing the temporal dependencies of the data by retaining historical state information.

[0117] In one implementation, the memory network module can be a recurrent neural network (RNN), a long short-term memory network (LSTM), a temporal convolutional network (TCN), or the like.

[0118] In one implementation, temporal feature data is used as a numerical representation to reflect the dynamic changes in video data, including patterns in the time dimension. For example, this temporal feature data can be used to indicate head movement trends.

[0119] S102112. Input the vehicle control data into the encoding module in the arbitration model to convert the vehicle control data into the corresponding encoded feature data.

[0120] For example, the controller can input the acquired vehicle control data into the encoding module of the arbitration model. This encoding module converts the original control signal into a numerical feature vector through feature engineering or deep learning, eliminating data heterogeneity.

[0121] In one implementation, the encoding module is a sub-network in the arbitration model responsible for data preprocessing, used to convert unstructured or heterogeneous control signals into structured feature representations.

[0122] In one implementation, the encoded feature data is a normalized numerical vector.

[0123] In one implementation, the encoding module can use regularized encoding to apply one-hot encoding or label encoding to discrete control signals. Alternatively, it can normalize or quantize continuous control signals. For example, the value can be scaled to between 0 and 1. Or, the value can be divided into 5 levels.

[0124] In one implementation, the encoding module can map a high-dimensional discrete signal into a low-dimensional dense vector through an embedding layer using embedding encoding.

[0125] In one implementation, the encoding module can use an autoencoder to compress control data using unsupervised learning and generate dimensionality-reduced features through an encoder-decoder structure.

[0126] S102113. Based on the user's historical behavior data and voice data, the semantic feature data of the user's historical behavior data and voice data are extracted using the perceptual network module in the arbitration model.

[0127] For example, the controller integrates user historical behavior data and voice data, and inputs them into the perceptual network module of the arbitration model. This module extracts semantic features through a multimodal fusion mechanism to obtain semantic feature data.

[0128] In one implementation, user historical behavior data can be records of the user's past 10 operations.

[0129] In one implementation, the perceptual network module is a sub-network in the arbitration model responsible for understanding the semantics of the data and extracting abstract features by combining contextual information.

[0130] In one implementation, semantic feature data is used as a numerical representation reflecting user intent or behavioral patterns.

[0131] In one implementation, the perception network module can extract the temporal semantics of the operation sequence from historical behavioral data using networks such as Long Short-Term Memory (LSTM) and Transformer.

[0132] In one implementation, the perceptual network module can process speech data and extract textual semantics of instructions from pre-trained models such as Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT).

[0133] In this example, multimodal data hierarchical feature extraction is used to achieve deep integration and utilization of multimodal information, thereby improving data effectiveness.

[0134] In one example, in step S102111 above, the controller extracts the temporal feature data of the driver's head video data using the memory network module in the arbitration model, specifically including:

[0135] S1021111: Identify the pupil position at each moment in the driver's head video data to obtain the eye-tracking data sequence.

[0136] For example, the controller acquires video data of the driver's head using an in-vehicle camera and uses computer vision algorithms to detect the pupil center coordinates in each frame. The controller can then use these pupil center coordinates to generate an eye-tracking data sequence.

[0137] In one implementation, the pupil position is the pixel coordinate of the pupil center in the image. The GIA pupil position can be used to further determine the driver's gaze focus.

[0138] In one implementation, the eye-tracking data sequence is a set of pupil positions at consecutive time points, reflecting the dynamic changes in gaze.

[0139] In another implementation, the eye-tracking data sequence is a set of the driver's gaze focus positions at consecutive time points, reflecting the dynamic changes in gaze.

[0140] In one implementation, the eye-tracking data sequence can be denoted as: .

[0141] S1021112. Identify the facial expressions at each moment in the driver's head video data to obtain the expression data sequence.

[0142] For example, the controller performs facial detection on video frames to extract geometric features or texture changes in areas such as eyebrows, eyes, and mouth. The controller can then identify expression categories using a classification model and generate a time-ordered sequence of expression labels.

[0143] In one implementation, the adjudication model can have multiple pre-set expression tags. The controller can determine the expression tag corresponding to the user at the current moment by classifying facial expressions.

[0144] In one implementation, the facial expression data sequence may include a set of facial expression tags corresponding to facial expressions at various times in time.

[0145] Optionally, in the tag data sequence, the time difference between the time points corresponding to two adjacent emoji tags can be a preset value.

[0146] S1021113. Identify the head posture at each moment in the driver's head video data to obtain the posture data sequence.

[0147] For example, the controller detects head key points in video frames, estimates the head rotation angle relative to the camera through geometric transformation or a deep learning model, and generates a time-ordered sequence of head pose data. For instance, this pose data sequence can be denoted as...

[0148] In one implementation, head pose can include the rotation angle of the head in three-dimensional space. This head pose can be used to reflect the direction of gaze or body state.

[0149] In one implementation, the attitude data sequence is a set of head attitudes at consecutive time points. Any two adjacent data points have a fixed time difference. In another implementation, this time difference can be a preset value.

[0150] S1021114. Using the memory network module in the arbitration model, the eye-tracking data sequence, facial expression data sequence, and pose data sequence are processed respectively to obtain temporal feature data.

[0151] For example, the controller inputs sequence data of eye movements, facial expressions, and postures into the memory network module of the arbitration model, extracts dynamic patterns from the data through temporal modeling, and generates fused temporal feature data.

[0152] In one implementation, time-series feature data is used as a numerical representation to reflect the dynamic changes of multimodal data.

[0153] In this example, a holographic representation of the driver's state is achieved through multimodal temporal data hierarchical analysis and memory network collaborative modeling, thereby improving data utilization and data effectiveness.

[0154] In one example, in step S10212 above, the modal feature data of each modality are input into the fusion layer of the arbitration model to obtain the user's behavioral feature vector, specifically including:

[0155] S102121. Employing a multi-head attention mechanism, the first fused feature is obtained by aligning and fusing the modal feature data of each modality based on the temporal information of each modality.

[0156] For example, the controller calculates the relevance weights of different modal features in the temporal dimension through a multi-head attention mechanism. Specifically, the feature sequence of each modality is used as the Query, Key, and Value input to the attention module, and the temporal synchronization pattern between modalities is captured through scaleddot-product attention. Finally, the features are weighted and fused to generate the first fused feature.

[0157] In one implementation, temporal information alignment can be achieved by dynamically adjusting the contribution of each modality feature on the time axis through attention weights, so that the fused features reflect the temporal synergy between modalities.

[0158] S102122. Using a cross-attention mechanism, the semantic information of the modal feature data of each modality is aligned and fused to obtain the second fused feature.

[0159] For example, the controller uses features from different modalities as the query and key for cross-attention, generates weights by calculating semantic similarity between modalities, and performs a weighted summation on the values ​​to generate a second fusion feature that reflects the semantic association across modalities.

[0160] In one implementation, the cross-attention mechanism allows features from different modalities to be used as query conditions for each other, dynamically learning semantic associations between modalities (such as the correspondence between "distraction" in facial expressions and "gaze wandering" in eye movements).

[0161] In one implementation, semantic information alignment can eliminate the semantic gap between modalities by matching the feature dimensions that express the same state (such as fatigue or distraction) in different modalities through attention weights.

[0162] In one implementation, the controller can obtain the second fusion feature by bidirectionally calculating the attention weights of mode A to mode B and mode B to mode A through symmetrical cross attention.

[0163] In one implementation, the controller can use asymmetric cross attention to fix one modality as Query and the other modalities as Key / Value to obtain the second fused feature.

[0164] In one implementation, the controller can construct modal features as graph nodes through graph-structured cross-attention, and learn the semantic edge weights between modalities through a graph attention network (GAT) to obtain the second fused feature.

[0165] S102123. Perform fusion processing on the first fusion feature and the second fusion feature to obtain the user's behavior feature vector.

[0166] For example, the controller can fuse the first fused feature and the second fused feature through concatenation, weighted summation or gating mechanism to generate the final user behavior feature vector.

[0167] In one implementation, the vector integrates temporal dynamics and semantic association information, and can be directly input into a classifier (such as SVM, fully connected network) or used for regression tasks.

[0168] In one implementation, the numerical representation of the behavioral feature vector is a compact vector of the user's state, containing joint information of multimodal temporal and semantic information, which is the core basis for subsequent decision-making.

[0169] In one implementation, the fusion process can integrate features from different sources through a specific mechanism to solve the problems of inconsistent feature dimensions or information redundancy.

[0170] In this example, multimodal temporal-semantic dual-channel attention alignment and feature fusion are used to achieve full-dimensional dynamic characterization of user behavior features and improve the accuracy of state recognition, thereby enhancing data effectiveness.

[0171] In one example, step S103 above, triggering the confirmation or execution of the target function based on the target function and confidence level, specifically includes:

[0172] S1031. If the confidence level is greater than or equal to the preset threshold, the execution of the target function is triggered.

[0173] For example, the controller can obtain a confidence level for the target function. The controller can store a preset threshold. The controller can compare the confidence level with the preset threshold. If the confidence level is greater than or equal to the preset threshold, the controller can trigger the execution of the target function.

[0174] In one implementation, the preset threshold can be a fixed value. For example, the preset threshold can be 0.7, 0.8, 0.9, etc.

[0175] In one implementation, the preset threshold can be a value dynamically determined based on environmental factors. For example, the preset threshold could be 0.9 for highways, or 0.7 for urban roads.

[0176] In one implementation, the preset threshold can be a multi-level threshold. For example, different preset thresholds can be set for different scenario modes. For instance, 0.7 triggers a warning sound, and 0.9 triggers emergency braking.

[0177] S1032. If the confidence level is less than the preset threshold, the confirmation of the target function is triggered.

[0178] For example, when the controller detects that the confidence level is less than a preset threshold, the controller may not directly execute the function, but instead initiate a confirmation process.

[0179] In one implementation, the controller can obtain user feedback through a human-computer interaction interface to confirm the target function.

[0180] In one implementation, if the user does not respond or refuses within a specified time, the execution is abandoned.

[0181] In this example, the accuracy of function execution and the system security are improved by using confidence threshold grading.

[0182] In one example, the controller can also log identified anomalies and optimize the model. This process may include:

[0183] S104. If the confirmation result of the target function is cancellation, then obtain the first operation successfully performed by the user within the preset time period, and establish the association between user behavior data and operation.

[0184] For example, when the controller detects that the user's confirmation result for the target function is "cancel," the controller first records the current timestamp and continuously monitors the user's operation within a preset time period. If it detects that the user has successfully performed a valid operation, it extracts the timestamp and operation of that valid operation. The controller can establish a correlation between the user behavior data obtained in step S101 and the valid operation.

[0185] In one implementation, the confirmation result is "cancel," specifically indicating that the user explicitly refuses to trigger the target function through the interactive interface. Optionally, this refusal method can be a button press, voice, or other similar method.

[0186] In one implementation, the preset duration is the maximum allowed time interval from confirmation of cancellation to the associated operation. This preset duration is used to define the validity of the association between the operation and the behavior, avoiding the association with irrelevant operations. For example, the preset duration can be 10s, 20s, 30s, etc.

[0187] S105. Periodically prompt users to view and confirm the correlation between user behavior data and operations. Then, use the confirmed correlation between user behavior data and operations to optimize the arbitration model.

[0188] For example, the controller generates a correlation report at a preset period. The controller displays the correlation report to the user through a human-computer interaction interface. The user can confirm the accuracy of the correlation through interaction. After collecting the user's confirmation, the controller inputs the valid correlation data into the arbitration model, and optimizes the model parameters through supervised learning or reinforcement learning to improve the accuracy of subsequent decisions.

[0189] In one implementation, the controller can send relationship reports to the user at fixed time intervals. Optionally, the fixed time interval can be daily, weekly, etc.

[0190] In one implementation, after a user cancels the target function, the user will typically try to execute the intended function again through other means. Therefore, the valid actions subsequently performed by the user can be considered the actual functions that the user needed to execute when the control case was triggered initially. To avoid situations where the user's subsequent valid actions are unrelated to the actual functions that needed to be executed when the control case was triggered, the controller needs to provide a confirmation prompt.

[0191] In one implementation, the controller can optimize the model parameters of the arbitration model through optimized training of the model.

[0192] In one implementation, the controller can periodically optimize the arbitration model according to a preset optimization period. Typically, the duration of the optimization period is longer than the fixed time interval for confirming the association.

[0193] In this example, the arbitration model dynamically optimizes the user state judgment by means of behavior-operation correlation closed loop correction, thereby further improving the accuracy of the arbitration model and enhancing the personalization of the arbitration model.

[0194] In one example, the arbitration model can run directly in the car's controller. Alternatively, the car's controller can send the multimodal user behavior data to the cloud, where the cloud can use the arbitration model to process the data and obtain the target function and confidence score. The cloud can then return the target function and confidence score to the controller.

[0195] The controller or cloud platform that runs the arbitration model may include an input module, an AI inference module, a function triggering module, and a self-learning module.

[0196] The input module may include a DMS driver behavior perception module, a voice recognition module, a steering wheel input event capture module, a current vehicle infotainment system front-end application status monitoring module, and a user history behavior database.

[0197] In one implementation, the DMS driver behavior perception module can capture facial images of the driver using a camera, and then use an edge computing module to parse facial feature vectors locally in real time. These features can be shown in Table 1.

[0198] Table 1

[0199]

[0200] In one implementation, data from the DMS driver behavior perception module is collected in milliseconds and sent to the AI ​​inference module.

[0201] The AI ​​inference module may include a multimodal fusion engine, an arbitration model, a functional priority evaluation module, and a decision arbitrator. The arbitration model can be a deep neural network for intent recognition.

[0202] For example, when the button in the center of the steering wheel is associated with multiple functions, it can include: music playback / pause; answering phone calls; waking up the AI ​​assistant; switching navigation routes; pausing short videos, etc. If the user clicks this button, the AI ​​inference module will determine the intent as "switching navigation routes" based on the user's current expression ("confused"), their gaze being directed at the central control screen, the navigation activity in the foreground, and the voice context ("how to change routes"), and trigger the relevant operation. This eliminates the need to rely on scenario-based code stacking or menu traversal.

[0203] The function triggering module may include a function mapping call interface, a redundant candidate feedback mechanism, and a security protection mechanism. This security protection mechanism is used to implement confidence assessment and confirmation logic.

[0204] The self-learning module can include a user behavior data accumulation system, an online model fine-tuning engine, and an intent-operation feedback closed-loop system. This module also supports each user's actions, with data stored locally and used for model fine-tuning, achieving a self-learning and personalized optimization mechanism.

[0205] For example, when used for the first time, the adjudication model adopts a general strategy; after the user presses the "right" button in the "central control voice interface" multiple times to "pause voice broadcast", the self-learning module can record the behavior and perform label backpropagation fine-tuning; if a similar operating environment is detected again, the adjudication model will prioritize recognizing the intention with higher confidence.

[0206] In one implementation, each user undergoes independent model optimization, achieving personalized optimization of the adjudication model. This process can improve the model's recognition accuracy as the frequency of use of the adjudication model increases. Furthermore, this mechanism can further ensure the security of users' personal information through localized data privacy processing.

[0207] Figure 2 This is a schematic diagram of the arbitration model provided by the present invention, as shown below. Figure 2 As shown, the controller running this arbitration model can employ a multi-input fusion architecture. The arbitration model includes:

[0208] The input multimodal user behavior data may include user head orientation, eye movement features, facial expression encoding, historical usage frequency matrix, current vehicle system status features, current voice keyword context, and steering wheel button encoding.

[0209] The user's head orientation can be marked using three parameters: pitch, yaw, and roll. Eye movement features can include three parameters: gaze direction, gaze target, and eye movement frequency. Facial expression encoding can be obtained based on action unit recognition. The historical usage frequency matrix contains information such as the usage frequency, time period, and scenario of each function. Current vehicle system status features can include parameters such as the current page, active applications, and function availability. Steering wheel button encoding can include button ID, press duration, and press force.

[0210] In one example, the arbitration model may include an input layer, an encoding layer, a fusion layer, an intent classification layer, and a confidence output layer.

[0211] The input layer enables parallel access to multimodal inputs. It receives data from multiple sensor sources and normalizes it into a tensor format suitable for input models. The input modalities are shown in Table 2.

[0212] Table 2

[0213]

[0214] The encoding layer processes each modality input using a separate encoder. Each modality can correspond to an independent encoding layer. The encoding layer can independently extract and process its corresponding modality. Each input modality is encoded into a unified high-dimensional representation (embedding), giving it context-aware capabilities and temporal feature representation.

[0215] Time-series data can be fed into a bidirectional long short-term memory (Bi-LSTM) network or a 1D convolutional neural network (CNN) and long short-term memory (LSTM) network to extract temporal features. For example, eye movement, facial, and head data can be processed to obtain time-series data.

[0216] Discrete features can be obtained by feeding them into an embedding layer and then concatenating or adding a single MLP transformation layer. For example, this can be used to process information such as vehicle system status and button codes.

[0217] Among them, features such as contextual behavior and speech sentence vectors can enhance semantic perception capabilities through the Transformer Encoder structure.

[0218] After encoding, the encoding layer can output the output of all modalities as a vector sequence with a uniform format. For example, the format can be [batch, 30, 64].

[0219] The Fusion Layer can use the Transformer mechanism to perform temporal context fusion. This intent layer can cross-fuse feature vectors from various modalities to construct a unified "user current state representation".

[0220] The fusion strategy of this fusion layer can employ the Transformer multi-head attention mechanism to perform cross-attention between different modal time points. Alternatively, it can use cross-attention and self-attention structures to enhance semantic alignment between modalities. Finally, it uses time-wise pooling or the Transformer's [CLS] token as the global intent feature vector. The fusion output is a single intent representation vector. For example, the format could be [batch, 128].

[0221] The Intent Classification Layer outputs the target intent category through a fully connected + Softmax interface, which can map the fused user state representation to a specific intent type.

[0222] This intent classification layer may include a fully connected layer (Dense). This fully connected layer can be:

[0223] makefile

[0224] FC1: 128 → 64 → ReLU FC2: 64 → Number of classes (N) → Softmax

[0225] An example of the output of this fully connected layer could be:

[0226] cpp [

[0228] 0.03, / / Switch songs

[0229] 0.07, / / Volume adjustment

[0230] 0.81, / / Route switching (highest)

[0231] 0.05, / / Open the voice assistant

[0232] 0.04 / / Make a phone call ]

[0234] That is, based on the confidence level of the output, the target function can be determined to be route switching.

[0235] The confidence estimation layer is used to assist in outputting the confidence level of the intent judgment, which is then used to trigger the mechanism's decision. This confidence estimation layer can add a "confidence" score to each intent judgment to help determine whether to execute the intent or pop up a confirmation.

[0236] In one implementation, the confidence output layer can use the maximum value in the Softmax output as the confidence level.

[0237] In another implementation, the confidence output layer can be supplemented with an auxiliary branch network (confidencehead), and the confidence level can be estimated through self-supervised or multi-task learning. The structure of this auxiliary branch network is as follows:

[0238] FC1: 128 → 32 → ReLUFC2: 32 → 1 → Sigmoid

[0239] For example, the maximum confidence level output through this auxiliary branch network can be 0.87.

[0240] In one implementation, the auxiliary branch network is a fully connected neural network submodule, typically used for confidence output or subclassification tasks.

[0241] Among them, FC1: 128 → 32 → ReLU can include:

[0242] FC1 represents a fully connected layer 1. This FC1 can accept an input vector of length 128. For example, this input vector could be fused user behavior data. The FC1 can map this input vector to a hidden layer representation of length 32. The mathematical expression for this process can be written as:

[0243]

[0244] in, The input vector. This is the weight matrix. This is the bias. ReLU is the activation function (Rectified Linear Unit). This is the output vector.

[0245] Wherein, FC2: 32 → 1 → Sigmoid can include:

[0246] FC2 represents fully connected layer 2. This FC2 layer receives the 32-dimensional hidden vector from the previous step and maps it to a single numerical output. This numerical value represents the confidence score. Furthermore, the confidence score is normalized to [0,1] using the sigmoid function. The mathematical expression for this process can be written as:

[0247]

[0248] in, This is the weight matrix. For bias. , where is the confidence level. This is the Sigmoid function. Its calculation formula is:

[0249]

[0250] The final use of this confidence output layer can be to output a numerical value representing the model's confidence score for the "current predictive intent".

[0251] In one implementation, if the confidence level is higher than a certain threshold, the function is triggered directly; if it is lower than the threshold, a confirmation pop-up is displayed to avoid accidental operation.

[0252] For example, if the system determines that "the user's current intention may be to switch to the next song," with a confidence level of 87%, the "next song" function will be triggered immediately if the threshold of 75% is exceeded.

[0253] If the confidence level is insufficient, such as only 53%, the system will display a voice confirmation prompt: "Switch to the next song? Please confirm."

[0254] In one implementation, the controller can also optimize the model independently for each user, thereby improving the recognition accuracy of the arbitration model as the frequency of use increases.

[0255] In one implementation, the controller can also implement localized data privacy processing based on the localization of the model, further improving data security.

[0256] In one implementation, if the execution result does not match the user's intention, the system records the error after the user manually cancels the process, thereby achieving model optimization.

[0257] In one implementation, the controller can make the judgment logic and data links traceable through logging and problem backtracking, which facilitates later debugging and training set optimization.

[0258] Figure 3 The flowchart of the control intention operation provided by the present invention is as follows: Figure 3As shown, when the user clicks the steering wheel control button, the controller can activate the DMS sensor to collect data such as eye movement trajectory, facial expression, head posture, and operation duration / force. Additionally, the controller can simultaneously collect the current vehicle system status, such as the current application, page, background tasks, and card display status.

[0259] The controller can organize the above information into multimodal user behavior data. This multimodal user behavior data may include time series processing, discrete state one-hot encoding / embedding, and user context behavior statistics. The controller can input the user behavior data into the arbitration model and obtain a confidence score through the sigmoid output. This confidence score is between 0 and 1. If the confidence score is greater than or equal to a preset threshold, the controller can determine to directly trigger the function execution. Otherwise, if the confidence score is less than the preset threshold, the controller can prompt the user with a pop-up window / voice confirmation to confirm whether to execute the function. The user can confirm to execute the function, or the user can cancel to execute the function.

[0260] Figure 4 This is a schematic diagram of the vehicle steering control device provided by the present invention, as shown below. Figure 4 As shown, the vehicle steering control device 400 provided in this embodiment includes:

[0261] The acquisition module 401 is used to acquire multimodal user behavior data when the user triggers the control button.

[0262] The processing module 402 is used to input multimodal user behavior data and the function information of the control button into a preset arbitration model to obtain the target function and confidence level of the control button triggered by the user. Based on the target function and confidence level, the module triggers the confirmation or execution of the target function.

[0263] In one example, the function information of the control button includes multiple functions corresponding to the control button. Processing module 402 is used for:

[0264] An arbitration model is used to process multimodal user behavior data to obtain user behavior feature vectors.

[0265] The behavioral feature vector and the multiple functions corresponding to the control buttons are input into the fully connected layer of the arbitration model to obtain the confidence level of each function.

[0266] The function with the highest confidence level is identified as the target function.

[0267] In one example, processing module 402 is used for:

[0268] Using an arbitration model, the behavioral data of each modality in the user behavior data is normalized and encoded to obtain the modal feature data of each modality.

[0269] Modal feature data from each modality are input into the fusion layer of the arbitration model to obtain the user's behavioral feature vector.

[0270] In one example, the user behavior data for each modality includes driver head video data, vehicle control data, user historical behavior data, and voice data. Processing module 402 is used for:

[0271] Based on the driver's head video data, the temporal feature data of the driver's head video data is extracted using the memory network module in the arbitration model.

[0272] The vehicle control data is input into the encoding module in the arbitration model, and the vehicle control data is converted into corresponding encoded feature data.

[0273] Based on user historical behavior data and voice data, the semantic feature data of user historical behavior data and voice data are extracted using the perceptual network module in the arbitration model.

[0274] In one example, processing module 402 is used for:

[0275] The position of the pupils at each moment in the driver's head video data is identified to obtain the eye-tracking data sequence.

[0276] The facial expressions at each moment in the driver's head video data are identified to obtain an expression data sequence.

[0277] The head posture at each moment in the driver's head video data is identified to obtain the posture data sequence.

[0278] The memory network module in the arbitration model is used to process the eye-tracking data sequence, facial expression data sequence, and pose data sequence respectively to obtain temporal feature data.

[0279] In one example, processing module 402 is used for:

[0280] A multi-head attention mechanism is adopted, and the first fused feature is obtained by aligning and fusing the modal feature data of each modality based on the temporal information of each modality.

[0281] A cross-attention mechanism is adopted to align and fuse the semantic information of the modal feature data of each modality to obtain the second fused feature.

[0282] The first and second fusion features are fused to obtain the user's behavioral feature vector.

[0283] In one example, processing module 402 is used for:

[0284] If the confidence level is greater than or equal to the preset threshold, the execution of the target function will be triggered.

[0285] If the confidence level is less than the preset threshold, the confirmation of the target function will be triggered.

[0286] In one example, processing module 402 is also used for:

[0287] If the confirmation result of the target function is cancellation, then obtain the first operation successfully performed by the user within the preset time period, and establish the association between user behavior data and operation.

[0288] Periodically prompt users to view and confirm the correlation between user behavior data and actions. Then, use the confirmed correlation between user behavior data and actions to optimize the arbitration model.

[0289] The vehicle steering control device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0290] Figure 5 This is a schematic diagram of the controller provided by the present invention. Figure 5 As shown, the controller 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0291] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0292] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0293] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0294] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0295] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0296] The present invention also provides an automobile, wherein the automobile is equipped with the above-described controller, which can execute the vehicle steering control method in the above embodiments.

[0297] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0298] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0299] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0300] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0301] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0302] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0303] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0304] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0305] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0306] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A vehicle steering control method, characterized in that, include: When a user triggers a control button, multimodal user behavior data is acquired; The multimodal user behavior data and the functional information of the control button are input into a preset arbitration model to obtain the target function and confidence level of the control button triggered by the user. Based on the target function and the confidence level, the confirmation or execution of the target function is triggered; The function information of the control button includes multiple functions corresponding to the control button; the multimodal user behavior data and the function information of the control button are input into a preset arbitration model to obtain the target function and confidence level of the control button triggered by the user, including: The arbitration model is used to process the multimodal user behavior data to obtain the user's behavior feature vector. The behavioral feature vector and the multiple functions corresponding to the control button are input into the fully connected layer of the arbitration model to obtain the confidence level of each function; The function with the highest confidence level is identified as the target function. The arbitration model is used to process the user behavior data of each modality to obtain the user's behavior feature vector, including: Using the arbitration model, the behavioral data of each modality in the user behavior data are normalized and encoded to obtain modal feature data for each modality; The modal feature data of each modality are input into the fusion layer of the arbitration model to obtain the user's behavioral feature vector; The user behavior data for each modality includes driver head video data, vehicle control data, user historical behavior data, and voice data. Using the arbitration model, the behavior data for each modality in the user behavior data is standardized and encoded to obtain modal feature data for each modality, including: Based on the driver's head video data, the temporal feature data of the driver's head video data is extracted using the memory network module in the arbitration model; The vehicle control data is input into the encoding module in the arbitration model, and the vehicle control data is converted into corresponding encoded feature data. Based on the user's historical behavior data and voice data, the semantic feature data of the user's historical behavior data and voice data are extracted using the perceptual network module in the arbitration model.

2. The method according to claim 1, characterized in that, Based on the driver's head video data, the temporal feature data of the driver's head video data is extracted using the memory network module in the arbitration model, including: Identify the pupil position at each moment in the driver's head video data to obtain an eye-tracking data sequence; The facial expressions at each moment in the driver's head video data are identified to obtain an expression data sequence; Identify the head posture at each moment in the driver's head video data to obtain a posture data sequence; The memory network module in the arbitration model is used to process the eye-tracking data sequence, the facial expression data sequence, and the pose data sequence to obtain the temporal feature data.

3. The method according to claim 1, characterized in that, The modal feature data of each modality are input into the fusion layer of the arbitration model to obtain the user's behavioral feature vector, including: A multi-head attention mechanism is adopted, and the modal feature data of each modality are aligned and fused based on the temporal information of each modality to obtain the first fused feature; A cross-attention mechanism is used to align and fuse the semantic information of the modal feature data of each modality to obtain a second fused feature; The first fusion feature and the second fusion feature are fused together to obtain the user's behavioral feature vector.

4. The method according to any one of claims 1-3, characterized in that, Based on the target function and the confidence level, triggering the confirmation or execution of the target function includes: If the confidence level is greater than or equal to a preset threshold, the execution of the target function is triggered; If the confidence level is less than the preset threshold, then the confirmation of the target function is triggered.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: If the confirmation result of the target function is cancellation, then obtain the first operation successfully performed by the user within the preset time period, and establish the association between the user behavior data and the operation; The system periodically prompts users to view and confirm the correlation between the user behavior data and the operation; and uses the confirmed correlation between the user behavior data and the operation to optimize the arbitration model.

6. A controller, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-5.

7. A car, characterized in that... include: The vehicle is equipped with the controller as described in claim 6.

Images