Dynamic vehicle machine HMI system based on multi-modal interaction

Abstract

The invention relates to the technical field of intelligent cabins, in particular to a dynamic vehicle machine HMI system based on multi-modal interaction. Comprising a multi-mode biological feature acquisition module, a feature fusion identity verification module, a driving state and split screen control analysis module, a voice interaction and role instruction processing module, a driver instruction prediction module and an interaction processing module without role recognition. Wherein the driver instruction prediction module receives a historical control instruction of each driver after successful driver identity verification, constructs a historical instruction sequence, and trains an instruction prediction model corresponding to each driver; receiving a role corresponding to the output voice signal, inputting a voice vector to a corresponding instruction prediction model, and generating a prediction control instruction; and when a prediction control instruction is generated, a joint loss function is designed, so that the instruction prediction model learns a personalized instruction mode of a current driver and a common instruction mode of multiple drivers at the same time.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit technology, and more specifically, to a dynamic vehicle-machine HMI system based on multimodal interaction. Background Technology

[0002] With the rapid development of automotive intelligence, the vehicle-machine interface (HMI) has become a key element in improving the driving experience and the level of vehicle intelligence. Current main HMI technologies have significant limitations: traditional touchscreen interaction requires the driver to operate functions through the central control screen, but this necessitates frequent shifts of the driver's gaze from the road to the screen, making the operation cumbersome, distracting, and posing significant driving safety hazards; while voice control allows operation of vehicle functions via voice commands, the voice recognition rate drops significantly in noisy environments (such as high-speed wind noise, tire noise, and engine noise) or when the driver speaks a dialect, leading to delayed or incorrect command responses. This necessitates the driver to repeat commands, indirectly causing driver distraction; physical buttons and knobs, although retaining some... While basic functions are operated separately, the button functions are fixed and cannot be flexibly expanded. They also occupy limited space in the cabin, affecting interior design and ease of use. Although split-screen display technology supports multi-tasking split-screen display (such as displaying navigation on the driver's side and entertainment content on the passenger's side), the split-screen layout is static and fixed. It cannot be dynamically adjusted according to driving scenarios (such as high-speed driving and low-speed parking) or driver status (such as fatigue or distraction). If only entertainment content is displayed on the passenger's side, the static split-screen interface may still attract the driver's attention to briefly deviate from the road, increasing the risk of distraction and failing to meet users' needs for convenient, efficient, and safe interaction. In today's fast-paced life, consumers' expectations for the functionality of car cabins are constantly rising. They are no longer satisfied with using cars as mere driving tools, but rather aspire to transform them into a "third living space" integrating travel, work, and entertainment. This necessitates that in-vehicle infotainment systems possess more powerful interactive capabilities to cope with diverse usage scenarios and user needs. Simultaneously, multimodal interaction technology is rapidly emerging in the field of human-computer interaction. Its ability to integrate multiple sensory channels such as voice, vision, and gestures for information interaction has brought new opportunities for the innovation of in-vehicle HMIs. However, even with features such as voice commands and touch operation, existing in-vehicle HMI systems still suffer from insufficient coordination in practical applications—during the driver's driving process, for... To avoid frequently shifting their gaze to operate the touchscreen, voice interaction is usually chosen. However, voice recognition is easily affected by environmental interference and there are deviations in the understanding of commands. Drivers need to repeatedly confirm or correct commands, which prolongs the time of distraction and leads to driving distraction. At the same time, although the multi-tasking split-screen display supported by the touchscreen can meet the entertainment and work needs of in-vehicle passengers (such as front passenger and rear passenger), and can achieve multi-functional parallel display during the split-screen process, the split-screen layout cannot dynamically adapt to the driving scenario. The static multi-area interface may still cause visual interference to the driver, causing the driver to unconsciously focus on unnecessary information, further increasing driving safety risks. In view of this, we propose a dynamic in-vehicle HMI system based on multimodal interaction. Summary of the Invention

[0003] The purpose of this invention is to solve the problems of current vehicle-mounted human-machine interaction systems, such as cumbersome operation that easily distracts the driver, voice recognition being easily affected by environmental interference leading to command response issues, fixed button functions that occupy space, and static split-screen layouts that cannot be dynamically adapted to driving scenarios, which affect driving safety and ease of interaction.

[0004] To achieve the above objectives, this invention provides a dynamic vehicle-mounted HMI system based on multimodal interaction, comprising a multimodal biometric feature acquisition module, a feature fusion authentication module, a driving state and split-screen control analysis module, a voice interaction and role command processing module, a driver command prediction module, and an interaction processing module for situations without role recognition, wherein: The multimodal biometric acquisition module establishes an interactive connection with the vehicle ECU via a CAN bus to acquire vehicle status data and uses sensor technologies such as in-vehicle camera-captured facial images and microphone array-captured real-time voice signals to collect real-time data. The feature fusion and identity verification module trains a multimodal biometric fusion model, fusing pupil and eye features output by the model as verification features to construct an identity feature template for verifying the driver's identity. Accurate driver identification provides a foundation for subsequent personalized driver-based interactions and controls, reliably verifying driver identity and ensuring the system provides adaptive services for different drivers. The driving status and split-screen control analysis module analyzes whether it is necessary to disable the passenger entertainment system. The split-screen control command can be turned off in appropriate scenarios to prevent entertainment content in the passenger seat from interfering with driving and improve driving safety. The voice interaction and role command processing module includes a voice vector and control command determination unit and a role recognition and command model invocation unit. The voice vector and control command determination unit receives the voice vector output by the multimodal biometric fusion model. The role recognition and command model invocation unit analyzes the voice signal to determine the direction of the sound source and determines the role corresponding to the output coordinates; constructs voice features and voice feature templates; establishes a mapping relationship between voice features and verification features; after a passenger other than the driver outputs a voice signal, it is determined whether it is the previous driver. If so, the corresponding control command is recorded synchronously. The role recognition and command model invocation unit is used to identify the source role of the voice signal, construct relevant feature templates and mapping relationships, record passenger control commands, accurately locate which role in the vehicle the voice comes from, improve the voice feature system, and facilitate the management and subsequent use of passenger commands. The driver command prediction module receives the historical control commands of each driver after successful driver authentication, constructs a historical command sequence, and trains the command prediction model corresponding to each driver; it receives the output voice signal corresponding to the role, inputs the voice vector to the corresponding command prediction model, and generates predicted control commands; and when generating predicted control commands, it designs a joint loss function so that the command prediction model can simultaneously learn the personalized command pattern of the current driver and the common command pattern of multiple drivers. If the role recognition and instruction model calling unit cannot determine the role corresponding to the output coordinates, the interaction processing module without role recognition receives the comparison result of vehicle speed and vehicle speed threshold in the vehicle status data. If the vehicle speed in the vehicle status data is less than or equal to the vehicle speed threshold, an inquiry instruction is output. Under appropriate vehicle speed conditions, the intention of the instruction can be clarified through inquiry, ensuring effective interaction between the vehicle system and the people in the vehicle.

[0005] Compared with known public technologies, the technical solution provided by this invention has the following beneficial effects: First, a multimodal biometric fusion model for outputting eye features, pupil features, and voice vectors is trained through the feature fusion model training unit. Further, the passenger-side entertainment split-screen closure judgment unit and the driver fatigue judgment unit can determine whether the passenger-side entertainment split-screen needs to be closed and whether the driver is fatigued based on real-time pupil and eye features. This allows for timely judgment and response to the passenger-side entertainment split-screen status and driver fatigue through accurate biometric recognition, improving safety and intelligent interaction during driving, better adapting to different driving scenarios and driver states, and providing users with a more convenient and safer in-vehicle interaction experience. After training the multimodal biometric fusion model, the driver identity verification unit, during driver registration, receives the authentication signal and fuses the pupil and eye features into verification features to construct an identity feature template. Subsequently, before driving the vehicle, the driver again fuses the current pupil and eye features into features to be verified, using the features to be verified and the identity feature template to verify the driver's identity. The voice vector and control command determination unit determines the control command corresponding to the voice vector based on the voice vector output by the multimodal biometric fusion model. After determining the control command, the output coordinates of the voice signal are further determined by the role recognition and command model invocation unit to analyze whether it is a voice signal output by the driver. The driver command prediction module constructs a historical command sequence for each driver's control commands and then trains a command prediction model for each driver. During the use of the command prediction model, since the driver's identity is first verified by the driver authentication unit, the command prediction model corresponding to the current driver can be invoked to output the predicted control command. Furthermore, the driver command prediction module described in this invention further assists in generating predicted control commands by using historical command sequences from other drivers. By leveraging a multimodal biometric fusion model to accurately acquire voice vectors and determine control commands, and by analyzing the voice source in conjunction with output coordinates, it can accurately identify the command initiator. By constructing a dedicated command prediction model based on historical command sequences and then using identity verification to retrieve the corresponding model, it can accurately predict the current driver's command. At the same time, by combining the assistance of historical command sequences from other drivers, it can integrate the advantages of group command patterns, improve the accuracy and adaptability of command prediction, and enable the vehicle system to respond more intelligently to the command needs of different drivers, enhancing the convenience and intelligence of interaction.

[0006] Meanwhile, when multiple passengers are present in the vehicle while the driver is driving, the role recognition and command model invocation unit can determine whether the corresponding role is a previous driver based on the current voice signal. If it is a previous driver, the corresponding command prediction model is directly invoked; if not, feedback is given to the current driver to determine whether authorization is granted. If authorization is granted, and since no corresponding command prediction model exists for the authorized role, each command prediction model generates a predictive control command based on its learned command pattern, and then the command with the largest proportion is invoked for output. This allows for efficient use of the previous driver's command prediction model, rapid response to the previous driver's command needs, and improved interaction efficiency. On the other hand, when facing a newly authorized role without a dedicated model, generating and selecting the command with the largest proportion through multiple models can combine the characteristics of the group command pattern to improve the accuracy and reliability of command prediction, ensure the effective response of the vehicle system to various role commands, and enhance the intelligence and adaptability of vehicle-machine interaction. Attached Figure Description

[0007] Figure 1 This is a diagram of the architecture of the present invention; Figure 2 This is a flowchart of the dynamic split-screen process of the present invention.

[0008] The meanings of the labels in the diagram are as follows: 1. Multimodal biometric feature acquisition module; 2. Feature fusion and identity verification module; 21. Feature fusion model training unit; 22. Driver identity verification unit; 3. Driving status and split-screen control analysis module; 31. Passenger entertainment split-screen closing judgment unit; 32. Driver fatigue driving judgment unit; 4. Voice interaction and role command processing module; 41. Voice vector and control command determination unit; 42. Role recognition and command model calling unit; 5. Driver command prediction module; 6. Interaction processing module when there is no role recognition. Detailed Implementation

[0009] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] refer to Figures 1-2 As shown, a dynamic vehicle-mounted HMI system based on multimodal interaction includes a multimodal biometric feature acquisition module 1, a feature fusion and identity verification module 2, a driving state and split-screen control analysis module 3, a voice interaction and role command processing module 4, a driver command prediction module 5, and an interaction processing module 6 for situations without role recognition, wherein: The multimodal biometric acquisition module 1 establishes an interactive connection with the vehicle ECU via a CAN bus to acquire vehicle status data. The specific working principle is as follows: the vehicle ECU collects vehicle status data in real time (e.g., the transmission ECU collects gear position, the BMS collects remaining battery power, and relevant ECUs collect vehicle speed), and encapsulates the data into CAN data frames containing an ID field (uniquely identifying the data type, such as a specific ID corresponding to vehicle speed or gear position), a DLC field (indicating data segment length), and a data segment (storing actual vehicle status data in binary format) according to the vehicle CAN protocol specification. The CAN bus transmits differential signals via twisted-pair cable (to resist vehicle electromagnetic interference). The multimodal biometric acquisition module 1 receives and parses the CAN data frames, parses the data segments according to a preset ID-data mapping rule, and converts the binary information into vehicle status data. The multimodal biometric acquisition module 1 also uses sensor technologies such as in-vehicle camera-collected facial images and microphone array-collected voice signals to collect real-time data.

[0011] The feature fusion authentication module 2 includes a feature fusion model training unit 21 and a driver authentication unit 22. The feature fusion model training unit 21 trains a multimodal biometric fusion model. This model receives facial images captured by an in-vehicle camera and outputs corresponding pupil and eye features from the facial images. It also receives real-time speech signals from a microphone array and outputs speech vectors, specifically multimodal data (including facial images and their eye features, such as the coordinates of the left eye corner) that has been extensively annotated. Left midline coordinate of upper eyelid Right midline of upper eyelid Coordinates of the right eye corner Lower eyelid right midline coordinate Lower eyelid left midline coordinate Pupil features (pupil center coordinates) Inner eye coordinates Outer eye coordinates The multimodal data (speech signal and its corresponding speech vector) is divided into training set and validation set. A multimodal biometric fusion model is trained. When a face image is input, the multimodal biometric fusion model outputs the pupil feature and eye feature (the above coordinates) of the face image. When a speech signal is input, the multimodal biometric fusion model outputs the speech vector corresponding to the speech signal. The aforementioned large amount of labeled multimodal data can be obtained through several methods: Firstly, basic facial image and speech signal data can be obtained from public datasets, such as the Labeled Faces in the Wild (LFW) face database and the TIMIT speech corpus, and then supplemented with annotations of eye features (coordinates of the left eye corner) using specialized tools. Left midline coordinate of upper eyelid Right midline of upper eyelid Coordinates of the right eye corner Lower eyelid right midline coordinate Lower eyelid left midline coordinate Pupil features (with pupil center coordinates) Inner eye coordinates Outer eye coordinates ) and speech vectors; On the other hand, it includes, but is not limited to, real-time acquisition of facial images from in-vehicle cameras during the use of the multimodal biometric fusion model, and facial images of people who have been successfully identified; the output of the multimodal biometric fusion model is the speech vector in the speech signal, and the speech vector that has been successfully matched with the template vector. Then, the multimodal biometric fusion model can be iteratively optimized using the speech signal that has been successfully matched with the speech vector, the facial image of the person who has been successfully identified, the pupil features, and the eye features.

[0012] The working principle of the feature fusion model training unit 21 for training multimodal biometric feature fusion models is as follows: For face images in the training set... ( For face images Height (in pixels), The width (in pixels) of the input. The number of input channels (e.g., RGB image) The predicted coordinates are extracted using a multi-layer convolutional neural network (CNN), which includes convolutional layers, pooling layers, and fully connected layers. Convolutional layers are created by setting different convolutional kernels. ( The kernel size is [size]. (Number of convolution kernels), convolution kernel Using a fixed step size (e.g., step size 1 or 2) in the face image Swipe up, each swipe corresponds to a face image Element-wise multiplication and summation are performed on local regions, and the result is used as the output feature map. The corresponding pixel values ​​are used to slide the entire face image through each convolutional kernel. Then, feature maps are generated. Feature map Each pixel value on the feature map indicates whether the location contains the feature corresponding to that convolutional kernel (the larger the value, the more significant the feature). The position above and the Each convolutional kernel has a pixel value. for: ; in For the first Each convolutional kernel is located at... Weight at point For face images The pixel value at the corresponding position, For bias terms (adjusting feature maps) The overall offset, with each convolutional kernel corresponding to a bias). For activation functions; Pooling layers affect the output feature maps of convolutional layers. Downsampling is performed to reduce the size of the feature maps, thereby reducing the number of parameters in the multimodal biometric feature fusion model and lowering computational complexity. Specifically, the pooling layer sets a pooling window (the pooling window size is...). The maximum value within a local region of the pooling window is selected as the feature representative, and the specific max pooling layer outputs a feature map. In position , passageway The element value is: ; in To output feature map Location index, For channel indexing, For feature map The element value at the corresponding position and in the corresponding channel; After multiple convolutional and pooling operations, a feature map containing eye and pupil features is obtained. The fully connected layer flattens the feature map containing eye and pupil features, converting the two-dimensional feature map into a one-dimensional feature vector. ( (where is the dimension of the feature vector, i.e., the number of features after flattening), and then the feature vector... Predicted coordinates mapped to eye and pupil features ,in This is the weight matrix of the fully connected layer. To output the number of predicted coordinates; Bias for fully connected layers; For the speech signals in the training set First, convert the speech signal. For the spectrum diagram: specifically, the speech signal Divided into lengths of The short time frame, for each frame of signal ( Multiplied by Hanning window ( , (frame length), for speech signals Perform frame-by-frame windowing: ; After windowing each frame, the first Frame signal conduct Point Fourier Transform: (in , The imaginary unit, For the first Frame signal At frequency point Output the complex amplitude of the frame at different frequency points (including amplitude and phase information): ,in and They are respectively The real and imaginary parts; taking the logarithm of the amplitude spectrum, we get... This converts the speech signal. Logarithmic power spectrum (Related to pixel values ​​in the spectrogram); Again, extract the face image. Extracting Log-Power Spectrum from Multilayer Convolutional Neural Networks (CNNs) with High-Performance Features The local frequency features are used to output a fixed-dimensional predicted speech vector. ; After each training round, the multimodal biometric fusion model is evaluated using a validation set: the validation set (specifically, a face image or speech signal) is input to the multimodal biometric fusion model, and forward propagation yields the prediction results (eye features, pupil features, and speech vectors); the mean squared error (MSE) loss function is used to calculate the MSE loss. ,in Multimodal data in the validation set. For the first Predicted values ​​for multimodal data (specifically, predicted coordinates of eye and pupil features). Or predict speech vectors ), For the first The true values ​​of multimodal data (specifically, the true coordinates of eye and pupil features or the true speech vector) are used to iteratively update hyperparameters through backpropagation. A loss threshold is set, and the MSE loss is less than the loss threshold, thereby completing the training of the multimodal biometric fusion model.

[0013] Meanwhile, during the training of the multimodal biometric fusion model by the feature fusion model training unit 21, if the multimodal biometric fusion model performs well in the training and validation phases, a test set can be introduced to verify whether the final performance of the multimodal biometric fusion model meets expectations. The test set can also be used to determine the accuracy of the multimodal biometric fusion model in tasks such as feature prediction of new face images and speech signals.

[0014] The driver identity verification unit 22 senses the authentication signal (i.e., the driver registration process), receives the pupil features and eye features output by the multimodal biometric fusion model, connects the pupil features and eye features in a fixed order to fuse them into verification features, and constructs an identity feature template by using the verification features corresponding to different drivers. Before the driver drives the vehicle, the face image captured in real time by the in-vehicle camera is output to the multimodal biometric fusion model. The real-time pupil feature and real-time eye feature are fused as the feature to be verified. The verification similarity between the feature to be verified and multiple verification features in the identity feature template is calculated. A feature threshold is set. If the verification similarity is greater than the feature threshold, the identity verification is considered successful. Otherwise, a security warning signal is output.

[0015] The driving status and split-screen control analysis module 3 includes a passenger entertainment split-screen closure judgment unit 31 and a driver fatigue driving judgment unit 32. The passenger entertainment split-screen closure judgment unit 31 receives pupil features output by a multimodal biometric fusion model, using inner eye coordinates. Outer eye coordinates The line connecting the reference axes (horizontal direction) is used to calculate the center of the eye. Establish a two-dimensional eye coordinate system: the origin of the two-dimensional eye coordinate system is the center of the eye. , The axis points from the outer corner of the eye to the coordinates of the outer corner of the eye. ,, Axis perpendicular to Axial upward; Calculate the pupil center Two-dimensional offset in the eye coordinate system: Horizontal offset Horizontal offset A positive value indicates that the pupil is deflected outwards, while a negative value indicates that it is deflected inwards; Vertical offset Vertical offset A positive value indicates that the pupil is deflected upwards, while a negative value indicates that it is deflected downwards; Convert 2D offset to 3D line-of-sight vector: Line-of-sight horizontal angle vertical angle ,in The distance from the eyeball to the in-car camera. Coordinates of the inner corner of the eye Outer eye coordinates The actual physical distance is approximately 30-40mm, obtained through statistical values ​​or calibration. Focal length of the in-car camera; horizontal angle and vertical angle These represent the horizontal and vertical angles between the line of sight and the direction of direct vision, respectively. Preset angle threshold to divide the area, if the horizontal angle and vertical angle If any item is not within the corresponding angle threshold division area, it is determined that the driver's gaze has left the screen, and the duration of the gaze leaving the screen is recorded. ; Receive vehicle speed data from vehicle status data and set speed thresholds accordingly. Observation threshold If the vehicle speed in the vehicle status data >Speed ​​threshold and the duration of time away from the screen >Observation threshold Then output control commands. Used to disable the passenger-side entertainment split screen.

[0016] The driver fatigue judgment unit 32 receives real-time eye features and calculates the eye aspect ratio. ,in The Euclidean distance between the two coordinates; Set statistical time window Eyelid closure threshold Fatigue threshold If the length-to-width ratio of the eyes <Eyelid closure threshold Then, it is determined that the eyelid is closed, and then the time window is perceived. Total duration of inner eyelid closure Calculate the time window Percentage of time spent with inner eyelid closure If the time percentage ≥ Fatigue Threshold If the driver is fatigued, an inquiry template library is established (the inquiry template library includes the percentage of different eyelid closure times). (Inquiry voice prompts) Select the corresponding inquiry voice prompt for driver fatigue. Output; if time percentage < Fatigue Threshold If so, it is determined that the driver is driving normally.

[0017] The voice interaction and role instruction processing module 4 includes a voice vector and control instruction determination unit 41 and a role recognition and instruction model calling unit 42. The voice vector and control instruction determination unit 41 has a preset instruction library including multiple template vectors. The total number of template vectors not in the instruction library. For the first template vectors, template vector (dimensions), and each template vector Associate a control command ; Receive the speech vector corresponding to the speech signal output by the multimodal biometric fusion model Calculate speech vectors With each template vector in the preset instruction library Vector similarity between ,in For vector dot product, speech vectors The length of the mold, template vector The modulus length; Set vector threshold Cross-comparison of speech vectors With each template vector vector similarity Find the maximum similarity If the maximum similarity > Vector threshold Then retrieve the template vector. Associated control signals Output.

[0018] The role recognition and instruction model calling unit 42 receives the microphone array. (microphone array) The Middle The coordinates of the microphones are ( Real-time acquisition of audio signals; when audio signals are output from different sound sources, calculate the time difference in audio signal reception at each microphone: ; in For the first The coordinates of each microphone, For the first The coordinates of each microphone, The speed at which a speech signal travels through the air; Combined with microphone array The geometric layout determines the direction of the sound source, and the output coordinates of the speech signal are calculated. A specific calculation example is as follows: assuming a microphone array... It contains three microphones The corresponding time difference is Construct a system of nonlinear equations: ; The output coordinates of the speech signal can be obtained by numerical methods (such as the Newton-Raphson method). ; Define a spatial range for each role's corresponding area (each area corresponds to a different role, such as the driver's area, the front passenger area, different rear seat areas, etc.), and match and output coordinates. For each area inside the vehicle, determine the output coordinate matching area and call up the corresponding character for that area; If the output voice signal comes from the current driver, then multiple voice vectors corresponding to multiple voice signals (current voice signal and historical voice signal) of the same driver are continuously received, and the multiple voice vectors are fused into the driver's voice features: ,in To participate in the fusion of speech vectors Quantity, For the first speech vectors Weights in the fusion process (for each speech vector) All exist One dimension, namely ), speech vectors The Feature values ​​in each dimension; and record each speech vector. Corresponding control commands ; Establish speech features The mapping relationship between the verification features in driver authentication unit 22 (used to directly call up the corresponding instruction prediction model after successful driver authentication) and the voice features of multiple drivers. Together they form a voice feature template, enabling the driver authentication unit 22 to directly determine the current driver's corresponding historical control commands after successful driver authentication. When passengers other than the driver output their voice signals, the system receives the voice vector from the voice signal output by the multimodal biometric fusion model. A speech threshold is set, and the speech vector in the speech feature template is calculated by the speech vector and control command determination unit 41. With the current speech vector If the voice similarity is greater than the voice threshold, then the passenger at the current voice signal output coordinate is determined to be the historical driver, and the control command corresponding to each voice vector is recorded simultaneously. .

[0019] After successful verification, the driver command prediction module 5 receives the historical control commands of each driver and arranges them into a historical command sequence according to the time frame order. Based on the long short-term memory network, the historical command sequence of each driver is used as training data to train the command prediction model corresponding to each driver. Long Short-Term Memory (LSTM) networks analyze training data through forgetting gates, input gates, cell state updates, and output gates to capture long-term dependencies in historical instruction sequences. The Gate of Oblivion Information used to decide which cell state to discard, among which for Forget Gate Output It is the Sigmoid activation function. Here is the forget gate weight matrix. for Hide your status at all times. for Input at any time Forget gate bias term; Input gates include input gate outputs and candidate cell status ,in The input gate weight matrix, For the input gate bias term, The hyperbolic tangent activation function is used. This is the candidate cell state weight matrix. This refers to the candidate cell state bias term; Cell state update combines forget gate and input gate output ,in for The constantly updated state of the cell. for Cell state at any time For element-wise multiplication; The output gate first passes through Calculate the output of the output gate, and then pass through get Hide state at all times Used for subsequent predictions, among which for Output gate output at all times, This is the output gate weight matrix. This is the output gate bias term; When training the instruction prediction model, the mean squared error (MSE) is used as the loss function: ,in For the amount of training data, These are actual historical control commands. To predict control commands; and then update the weight matrix based on the optimization algorithm ( etc.) and bias terms ( (etc.) to minimize the loss function; It also receives the voice signal corresponding to the character from the output character recognition and instruction model calling unit 42, inputs the voice vector of the corresponding voice signal into the corresponding instruction prediction model, and generates a predictive control instruction output. When the predictive control model generates predictive control commands, it also utilizes control command sequences from different roles to assist in the generation of predictive control commands. Because different drivers may have similar command habits, using more historical data can enrich the model's learning samples and further improve prediction accuracy. The specific working principle is as follows: Multi-driver instruction feature extraction: For each historical driver's control instruction sequence, the long-term dependencies between control instructions are captured using the forget gate, input gate, cell state, and output gate in the Long Short-Term Memory network, and the feature vector of each driver's instruction sequence is output (the vector contains the core pattern of the corresponding driver's instructions (such as instruction type distribution, time triggering rules, etc.)). Cross-driver feature fusion training: When training the current driver's command prediction model, not only are the current driver's historical command sequences used, but also the command sequences and corresponding features of other historical drivers are introduced. Specifically, a joint loss function is designed so that the command prediction model can simultaneously learn the current driver's personalized command patterns and the common command patterns of multiple drivers. ; in , The current amount of driver training data, For the first Predicted values ​​for each training data set; , For the current driver's feature vector, For vector dot product, It is the 2-norm of the vector; To balance the hyperparameters of the two losses; For example, the joint loss function includes both the penalty for the current driver's prediction error and the constraint of the similarity between the current driver's features and those of other drivers. This prompts the instruction prediction model to learn common instruction knowledge across drivers. Specifically, an optimization algorithm is used to calculate the gradient of the joint loss function with respect to the parameters of the instruction prediction model (such as the weight matrix and bias terms). Based on the calculated gradient, the parameters (such as the weight matrix and bias terms) are updated according to the set learning rate. By repeatedly calculating the gradient and updating the parameters, the joint loss function is gradually reduced, thereby optimizing the parameters of the instruction prediction model. This allows the instruction prediction model to better learn both the current driver's personalized instruction patterns and the common instruction patterns of multiple drivers simultaneously.

[0020] Multi-source information fusion during prediction: When the current driver's speech vector is input into the corresponding command prediction model for prediction, the command prediction model generates a preliminary prediction result. Then, it combines common features learned from other drivers to correct the preliminary prediction result: the feature similarity between the current driver's features and those of other drivers is calculated. The higher the feature similarity between the driver and other drivers, the greater the influence weight of the driver's command pattern on the current prediction. Finally, multi-source information is fused to obtain a more accurate predictive control command. ; in Based on the Driver characteristics The generated predictive control commands (generated by the command prediction model of other drivers); For the current driver and the first Feature similarity weights for each driver (the higher the feature similarity, the greater the weight). The greater the driver's instruction pattern, the greater its impact on the prediction result for the current driver. Hyperparameters for controlling the influence of auxiliary information (tuned using a validation set).

[0021] If the voice signal analyzed and output by the role recognition and instruction model calling unit 42 is not from the driver, the interaction processing module 6 without role recognition receives real-time voice features. The output is sent to the role recognition and instruction model calling unit 42 to match the speech feature template. If the driver is a historical driver, the corresponding command prediction model of the historical driver is retrieved and output to the driver command prediction module 5. The driver command prediction module 5 then generates a predictive control command. If the role recognition and instruction model calling unit 42 analyzes and outputs a speech signal that does not match any of the speech features in the speech feature template (i.e., the speech similarity between the speech signal and all speech features in the speech feature template is ≤ speech threshold), the interaction processing module 6 without role recognition will receive the vehicle speed from the vehicle status data. With vehicle speed threshold The comparison results show that the vehicle speed in the vehicle status data... ≤ Vehicle speed threshold Then output the query command. To determine whether the current driver has authorized the vehicle, the system receives the current driver's voice vector and calculates the voice similarity between the current driver's voice vector and the corresponding voice vector in the voice feature template. If the voice similarity is greater than the voice threshold... If the authorization is successful, the voice vector of the authorized character is then input into the command prediction model of each historical driver. Each command prediction model generates a predictive control command based on the command pattern it has learned, and then the predictive control command with the largest proportion is output.

[0022] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dynamic vehicle-mounted HMI system based on multimodal interaction, characterized in that, It includes a multimodal biometric acquisition module (1), a feature fusion and identity verification module (2), a driving status and split-screen control analysis module (3), a voice interaction and role command processing module (4), a driver command prediction module (5), and an interaction processing module for situations without role recognition (6), wherein: The feature fusion authentication module (2) trains a multimodal biometric fusion model, integrates the pupil features and eye features output by the multimodal biometric fusion model as verification features, and constructs an identity feature template to verify the driver's identity; the driving status and split-screen control analysis module (3) analyzes whether a control command to close the co-driver's entertainment split screen needs to be output; the voice interaction and role command processing module (4) includes a voice vector and control command determination unit (41) and a role recognition and command model calling unit (42). The voice vector and control command determination unit (41) receives the voice vector output by the multimodal biometric fusion model; the role recognition and command model calling unit (42) analyzes the voice signal to determine the direction of the sound source and determines the role corresponding to the output coordinates; constructs voice features and voice feature templates; establishes a mapping relationship between voice features and verification features; after a passenger other than the driver outputs a voice signal, it is determined whether it is a historical driver. If so, the corresponding control command is recorded synchronously. The driver instruction prediction module (5) receives the historical control instructions of each driver after successful driver authentication, constructs a historical instruction sequence, and trains the instruction prediction model corresponding to each driver; receives the output voice signal corresponding to the role, inputs the voice vector to the corresponding instruction prediction model, and generates the predicted control instruction; and when generating the predicted control instruction, designs a joint loss function so that the instruction prediction model can learn the personalized instruction pattern of the current driver and the common instruction pattern of multiple drivers at the same time. If the role recognition and instruction model calling unit (42) cannot determine the role corresponding to the output coordinates, the interaction processing module (6) without role recognition receives the comparison result of vehicle speed and vehicle speed threshold in the vehicle status data. If the vehicle speed in the vehicle status data is less than or equal to the vehicle speed threshold, it outputs an inquiry instruction analysis to determine whether authorization is granted. When authorization is granted, the voice vector of the authorized role is input into the instruction prediction model of each historical driver. Each instruction prediction model generates a predictive control instruction based on the instruction pattern it has learned, and then calls out the predictive control instruction with the largest proportion for output.

2. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 1, characterized in that: The feature fusion authentication module (2) includes a feature fusion model training unit (21) and a driver authentication unit (22), wherein: the feature fusion model training unit (21) uses multimodal data to train a multimodal biometric fusion model, the multimodal biometric fusion model receives face images collected by an in-vehicle camera and outputs the corresponding pupil features and eye features in the face image; the multimodal biometric fusion model receives voice signals collected in real time by a microphone array and outputs voice vectors; the driver authentication unit (22) senses and establishes authentication signals corresponding to verification features, establishes an identity feature template including multiple verification features, constructs features to be verified before the driver drives the vehicle, and verifies the driver's identity.

3. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 2, characterized in that: The feature fusion model training unit (21) divides the multimodal data into a training set and a validation set. For the face images in the training set, it uses a multi-layer convolutional neural network containing convolutional layers, pooling layers and fully connected layers to extract pupil features and eye features. Convolutional layers slide across a face image with a fixed stride by setting different convolutional kernels, performing element-wise multiplication and summation with local regions of the face image, and generating output feature maps by combining bias terms and activation functions; pooling layers downsample the feature maps output by convolutional layers, and select the maximum value of a local region within the pooling window as the feature representative by setting a pooling window. After multiple convolutional and pooling operations, a feature map containing eye and pupil features is obtained. The fully connected layer flattens the two-dimensional feature map into a one-dimensional feature vector, and then maps the one-dimensional feature vector to eye and pupil features. For the speech signal in the training set, it is first segmented into short time frames. Each frame signal is multiplied by a Hanning window for frame-by-frame windowing. Then, an N-point Fourier transform is performed on each windowed frame signal to obtain complex amplitudes at different frequency points. The logarithm of the amplitude spectrum is taken and converted into a logarithmic power spectrum. A multi-layer convolutional neural network for extracting face image features is used to extract local frequency features of the logarithmic power spectrum and output a fixed-dimensional predicted speech vector. After each training round, the multimodal data in the validation set is input to the forward propagation of the multimodal biometric fusion model to obtain the prediction results of eye features, pupil features, and speech vectors. The mean squared error loss function is used to calculate the MSE loss between the predicted value and the true value. The hyperparameters are iteratively updated through backpropagation. A loss threshold is set. When the MSE loss is less than the loss threshold, the multimodal biometric fusion model is trained successfully.

4. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 3, characterized in that: The driver identity verification unit (22) senses and establishes the authentication signal corresponding to the verification feature, receives the pupil feature and eye feature output by the multimodal biometric fusion model, connects the pupil feature and eye feature in a fixed order to fuse them into the verification feature, and jointly constructs the identity feature template through the verification features corresponding to different drivers. Before the driver drives the vehicle, the face image captured in real time by the in-vehicle camera is output to the multimodal biometric fusion model. The real-time pupil feature and real-time eye feature are fused as the feature to be verified. The verification similarity between the feature to be verified and multiple verification features in the identity feature template is calculated. A feature threshold is set. If the verification similarity is greater than the feature threshold, the identity verification is considered successful. Otherwise, a security warning signal is output.

5. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 1, characterized in that: The driving state and split-screen control analysis module (3) includes a co-driver entertainment split-screen closing judgment unit (31) and a driver fatigue driving judgment unit (32). The co-driver entertainment split-screen closing judgment unit (31) uses the line connecting the inner and outer corner coordinates in the pupil feature as the horizontal reference axis to calculate the eye center. It establishes a two-dimensional eye coordinate system with the eye center, the X-axis pointing to the outer corner along the line connecting the eye corners, and the Y-axis perpendicular to the X-axis upward. It calculates the two-dimensional offset of the pupil center in the eye coordinate system, including horizontal and vertical offset. When the horizontal offset is positive, it means that the pupil is biased to the outside, and when it is negative, it is biased to the inside. When the vertical offset is positive, it means that the pupil is biased to the top, and when it is negative, it is biased to the bottom. It converts the two-dimensional offset into a three-dimensional gaze vector and calculates the horizontal and vertical angles of the gaze. The preset angle threshold divides the area. If either the horizontal or vertical angle is not within the corresponding angle threshold division area, it is determined that the driver's gaze has left the screen, and the duration of the gaze leaving the screen is recorded. Receive vehicle speed from vehicle status data, set vehicle speed threshold and observation threshold respectively. If the vehicle speed in the vehicle status data is greater than the vehicle speed threshold and the time away from the screen is greater than the observation threshold, output a control command to turn off the passenger entertainment split screen. The driver fatigue judgment unit (32) sets a statistical time window, an eyelid closure threshold, a fatigue threshold, and calculates the eye aspect ratio. If the eye aspect ratio is less than the eyelid closure threshold, the eyelid is judged to be closed. Then, the total duration of eyelid closure within the time window is sensed, and the proportion of eyelid closure within the time window is calculated. The total duration of eyelid closure within the time window is sensed, and the proportion of eyelid closure within the time window is calculated by dividing the total duration of eyelid closure by the statistical time window. If the proportion of time closure is greater than or equal to the fatigue threshold, the driver is judged to be fatigued, and an inquiry template library is established to retrieve the corresponding inquiry voice output for fatigued driving. If the proportion of time closure is less than the fatigue threshold, the driver is judged to be driving normally.

6. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 3, characterized in that: The speech vector and control command determination unit (41) has a preset instruction library including multiple template vectors. The preset instruction library includes multiple template vectors. It receives the speech vector corresponding to the speech signal output by the multimodal biometric fusion model, calculates the vector similarity between the speech vector and each template vector, sets a vector threshold, cross-compares the vector similarity between the speech vector and each template vector, and retrieves the maximum similarity. If the maximum similarity is greater than the vector threshold, the control signal associated with the template vector is retrieved and output. The role recognition and instruction model calling unit (42) receives the voice signal collected in real time by the microphone array. When different sound sources output voice signals, it calculates the time difference of each microphone receiving the voice signal, and constructs a nonlinear equation system by combining the geometric layout of the microphone array and the time difference. The output coordinates of the voice signal are obtained by solving the equations using numerical methods. Set a spatial range for the area corresponding to each character, match the output coordinates with each area inside the vehicle, determine the area where the output coordinates match, and call up the character corresponding to the area. If the output voice signal comes from the current driver, then the system continuously receives multiple voice vectors corresponding to multiple voice signals from the same driver, weights each voice vector by dimension, sums and fuses them into the driver's voice features, and records the control command corresponding to each voice vector. The role recognition and instruction model calling unit (42) is also used to establish a mapping relationship between voice features and verification features in the driver identity verification unit (22), and to construct a voice feature template by combining the voice features of multiple drivers; and when a passenger other than the driver outputs a voice signal, it receives the voice vector in the voice signal output by the multimodal biometric fusion model, sets a voice threshold, and calculates the voice similarity between the voice vector in the voice feature template and the current voice vector through the voice vector and control instruction determination unit (41). If the voice similarity is greater than the voice threshold, it determines that the passenger at the current voice signal output coordinate is the historical driver, and records the control instruction corresponding to each voice vector synchronously.

7. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 6, characterized in that: The driver instruction prediction module (5) receives the historical control instructions of each driver after successful verification, arranges the historical control instructions in the order of time frames to form a historical instruction sequence, and uses the historical instruction sequence of each driver as training data based on the long short-term memory network to train the instruction prediction model corresponding to each driver; the long short-term memory network analyzes the training data through the forget gate, input gate, cell state update and output gate; The forget gate uses the sigmoid activation function. The sigmoid activation function is used to concatenate the hidden state of the previous time step and the input of the current time step. Then, the concatenated result is multiplied by the forget gate weight matrix. Finally, the forget gate bias term is added to obtain the output of the forget gate at time t. The input gate consists of the input gate output and the candidate cell state. The previous hidden state and the current input are concatenated. The concatenated result is multiplied by the input gate weight matrix, and the input gate bias term is added. Then, the Sigmoid activation function is applied to obtain the input gate output. The cell state update involves multiplying the output of the forget gate element-wise with the cell state at the previous time step, and then multiplying the output of the input gate element-wise with the candidate cell state. The results of the two element-wise multiplications are then added together to obtain the updated cell state at time t. When training the command prediction model, the mean squared error is used as the loss function. First, the difference between the actual historical control command and the predicted control command in each training data is calculated and squared. Then, all squared differences are summed and divided by the number of training data to obtain the loss value. The weight matrix and bias term are updated according to the optimization algorithm to minimize the loss function.

8. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 7, characterized in that: When the role recognition and instruction model calling unit (42) analyzes and outputs the voice signal as the current driver, it inputs the current voice vector into the corresponding instruction prediction model to generate the predicted control instruction output. When the current driver's corresponding instruction prediction model generates the predicted control instruction, the driver instruction prediction module (5) simultaneously uses the control instruction sequence of the historical driver to assist in generating the predicted control instruction. For each historical driver's control command sequence, the long-term dependencies between historical command sequences are captured using the forget gate, input gate, cell state, and output gate in a long short-term memory network, and the feature vector of each driver's historical command sequence is output. When training the current driver's command prediction model, the command sequences and corresponding features of other historical drivers are introduced. Specifically, a joint loss function is designed so that the command prediction model can learn the current driver's personalized command pattern and the common command pattern of multiple drivers at the same time. The joint loss function includes prediction loss and matching loss: calculate the sum of squares of the differences between the feature vectors corresponding to the actual control commands and the predicted control commands in the current driver training data, and then divide by the number of training data to obtain the prediction loss; Calculate the dot product of the current driver's feature vector and the feature vectors of other drivers, then divide by the 2-norm product of the two. Summate all the results, invert them, and divide by the number of drivers to obtain the matching loss. Use an optimization algorithm to calculate the gradient of the joint loss function with respect to each parameter of the instruction prediction model. Based on the calculated gradient, update the parameters according to the set learning rate. By repeatedly calculating the gradient and updating the parameters, the joint loss function is gradually reduced, thereby optimizing the parameters of the instruction prediction model.

9. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 8, characterized in that: The driver instruction prediction module (5) inputs the current driver's voice vector into the corresponding instruction prediction model. The instruction prediction model generates a preliminary prediction result and corrects the preliminary prediction result by combining the common features learned from other drivers. First, it calculates the feature similarity between the current driver's features and the features of other drivers. The higher the feature similarity, the greater the influence weight of the driver's instruction pattern on the current prediction result. The weight is obtained by dividing the dot product of the current driver's feature vector and the feature vector of the kth driver by the norm product of the two. The predicted control instruction generated based on the kth driver's features is multiplied by the corresponding feature similarity weight and summed, then divided by the number of drivers, and then multiplied by the hyperparameter of the degree of influence of control auxiliary information. Finally, the result is added to the preliminary prediction result, and finally, multi-source information is fused to obtain a more accurate predicted control instruction.

10. The dynamic vehicle-mounted HMI system based on multimodal interaction according to claim 6, characterized in that: The role recognition and instruction model calling unit (42) analyzes and outputs a voice signal that does not come from the driver. The interaction processing module (6) without role recognition receives real-time voice features and outputs them to the role recognition and instruction model calling unit (42) to match whether the voice feature template is a historical driver. If it is a historical driver, the instruction prediction model corresponding to the historical driver is called up and output to the driver instruction prediction module (5). The driver instruction prediction module (5) generates a prediction control instruction. When the voice feature template does not contain the driver, the interaction processing module (6) without role recognition receives the comparison result of vehicle speed and vehicle speed threshold in the vehicle status data. If the vehicle speed in the vehicle status data is less than or equal to the vehicle speed threshold, it outputs an inquiry command to the current driver to ask whether to authorize. It receives the current driver's voice vector and calculates the voice similarity between the current driver's voice vector and the voice vector corresponding to authorization in the voice feature template. If the voice similarity is greater than the voice threshold, it determines that the authorization is successful. Then, it outputs the corresponding control command to the driver command prediction module (5), and the driver command prediction module (5) generates a predicted control command.

Images