Intelligent cabin interaction method and system and medium

Abstract

The invention relates to an intelligent cabin interaction method and system and a medium, and relates to the field of intelligent cabins, and the method comprises the steps: obtaining the facial expression and voice information of a user; performing sentiment classification on the facial expression and the voice information by using a deep learning model; based on the sentiment classification result, adjusting the environment atmosphere in the cabin; creating a three-dimensional holographic image on the vehicle-mounted head-up display through holographic projection equipment; receiving a voice instruction, a gesture action or a staring direction of a user; and controlling functions in the cabin according to the voice instruction, the gesture action or the staring direction. The user interaction experience, the emotion response ability and the intelligent level of the intelligent cabin can be improved, and safer, more comfortable and more personalized driving experience is provided for a driver.

Description

Technical Field

[0001] The present invention relates to the field of intelligent cockpits, and in particular to an intelligent cockpit interaction method, system and medium. Background Art

[0002] With the rapid development of intelligent vehicle technology, users' expectations for vehicle cockpits have not been limited to the basic driving environment. Modern drivers and passengers increasingly hope to obtain a more immersive, intelligent and emotional riding experience. Although the current intelligent cockpit systems meet users' basic needs to a certain extent, there is still room for improvement in many aspects.

[0003] Existing intelligent cockpit systems often rely on traditional methods such as voice and touch in human-computer interaction. These methods may not be intuitive and efficient enough in some complex or emergency situations. At the same time, the current systems also have deficiencies in emotional recognition and response capabilities, making it difficult to accurately understand users' emotional states and make corresponding adjustments. In addition, traditional information display methods may not provide users with a sufficiently immersive visual experience.

[0004] The personalization degree and learning and adaptation capabilities of intelligent cockpit systems are also aspects worthy of attention. Each user has unique preferences and needs, and existing systems may be difficult to fully adapt to this diversity. At the same time, systems lacking continuous learning and optimization capabilities may not be able to continuously improve their service quality over time. Therefore, it is of great significance to explore new technologies and methods to improve the performance and user experience of intelligent cockpit interaction. Summary of the Invention

[0005] In view of the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide an intelligent cockpit interaction method, system and medium, which can improve the user interaction experience, emotional response capabilities and intelligent level of the intelligent cockpit, and provide a safer, more comfortable and personalized driving experience for drivers.

[0006] To achieve the above object, the present invention adopts the following technical solutions.

[0007] In the first aspect, an intelligent cockpit interaction method provided by the present invention adopts the following technical solutions: Obtain the facial expressions and voice information of the user; Use a deep learning model to perform emotional classification on the facial expressions and voice information; Based on the emotional classification results, adjust the environmental atmosphere in the cockpit; Create a three-dimensional holographic image on the vehicle head-up display through a holographic projection device; Receive the voice commands, gesture actions or gaze directions of the user; And, control functions in the cockpit according to the voice command, gesture movement or gazing direction.

[0008] Furthermore, in the above intelligent cockpit interaction method, the obtaining of the user's facial expression and voice information includes: Collect the user's facial images through an in-vehicle camera, preprocess the facial images, and extract facial features; Collect the user's voice signals through an in-vehicle microphone array, preprocess the voice signals, and extract voice features.

[0009] Furthermore, in the above intelligent cockpit interaction method, the use of a deep learning model to perform emotion classification on facial expressions and voice information includes: Use a preset convolutional neural network to perform emotion classification on the facial expressions; Use a preset recurrent neural network to perform emotion classification on the voice information.

[0010] Furthermore, in the above intelligent cockpit interaction method, the adjustment of the environmental atmosphere in the cockpit includes: Adjust the lighting brightness and color in the cockpit; and / or Adjust the music type and volume in the cockpit; and / or Adjust the air conditioning temperature and wind speed in the cockpit.

[0011] Furthermore, in the above intelligent cockpit interaction method, the creation of a three-dimensional holographic image on the in-vehicle head-up display through a holographic projection device includes: Generate a holographic image using a seven-dimensional holographic projection algorithm; Project the holographic image onto the in-vehicle head-up display.

[0012] Furthermore, in the above intelligent cockpit interaction method, the seven-dimensional holographic projection algorithm includes: Capture images from multiple positions; Combine three-dimensional position, two-dimensional angle, time, and light attribute information to generate a holographic image.

[0013] Furthermore, in the above intelligent cockpit interaction method, the receiving of the user's voice command, gesture movement or gazing direction includes: Use a hidden Markov model for speech recognition; Use a fast object detection algorithm for gesture recognition; Use the pupillary corneal reflex method for eye tracking.

[0014] Furthermore, in the above intelligent cockpit interaction method, it further includes: Use a support vector machine algorithm to learn the user's interaction behavior; Optimize the control strategy of the cockpit function according to the learning results.

[0015] Furthermore, in the above intelligent cockpit interaction method, it further includes: Provide personalized navigation, entertainment, and communication services according to the user's historical interaction data and preferences.

[0016] In a second aspect, an intelligent cockpit interaction system provided by the present invention adopts the following technical solution: An emotion acquisition module, at least used to acquire the user's facial expressions and voice information; An emotion classification module, at least used to perform emotion classification on the facial expressions and voice information by using a deep learning model; An atmosphere adjustment module, at least used to adjust the environmental atmosphere in the cockpit based on the emotion classification result; A holographic projection module, at least used to create a three-dimensional holographic image on the in-vehicle head-up display through a holographic projection device; An instruction recognition module, at least used to receive the user's voice instructions, gesture actions, or gaze directions; And a function control module, at least used to control the functions in the cockpit according to the voice instructions, gesture actions, or gaze directions.

[0017] Furthermore, in the above intelligent cockpit interaction system, it further includes: An optimization training module, at least used to learn the user's interaction behavior by using a support vector machine algorithm, and optimize the control strategy of the cockpit function according to the learning results.

[0018] In a third aspect, a readable storage medium provided by the present invention adopts the following technical solution: A readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, they implement the intelligent cockpit interaction method as described in any one of the first aspects above.

[0019] In summary, compared with the prior art, the present invention includes at least one of the following beneficial technical effects: The present invention improves driving safety by timely detecting the driver's fatigue or abnormal emotional state through emotion recognition; enhances the user experience by automatically adjusting the cockpit environment according to the emotional state to create a comfortable atmosphere; simplifies the operation by using a multi-modal interaction method to make cockpit control more intuitive and convenient; provides personalized services by providing customized functions and content according to the user's habits and preferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those skilled in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of a specific embodiment of an intelligent cockpit interaction method of the present invention.

[0022] Figure 2 It is a schematic structural diagram of a specific embodiment of an intelligent cockpit interaction system of the present invention. Detailed implementation manners

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application. In addition, it should be understood that the specific implementation manners described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0024] It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments of the present application. And in the following embodiments, each embodiment has its own emphasis. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0025] For the method steps described in the embodiments of the present invention, their execution order can be executed in the order described in the specific implementation manner, or can be adjusted according to actual needs on the premise of being able to solve the technical problem. The execution orders are not listed one by one here.

[0026] Referring to Figure 1 , an intelligent cockpit interaction method provided by an embodiment of the present invention includes the following steps.

[0027] S1, obtain the facial expression and voice information of the user. Specifically, this step can collect the facial image of the user through an in-vehicle camera and collect the voice signal of the user through an in-vehicle microphone array. The collected facial image and voice signal can be preprocessed to extract facial features and voice features.

[0028] S2. Use a deep learning model to perform emotion classification on facial expressions and voice information. Specifically, in this step, a pre-trained convolutional neural network can be used to classify facial expressions, and a recurrent neural network can be used to classify voice information. The classification results can include positive emotions, negative emotions, neutral emotions, etc.

[0029] S3. Based on the emotion classification results, adjust the environmental atmosphere inside the cockpit. Specifically, in this step, parameters such as the lighting brightness and color, music type and volume, air conditioning temperature and wind speed inside the cockpit can be automatically adjusted according to the recognized user emotion state to create an environmental atmosphere that matches the user's emotion.

[0030] S4. Create a three-dimensional holographic image on the in-vehicle head-up display through a holographic projection device. Specifically, in this step, a seven-dimensional holographic projection algorithm can be used to generate a holographic image and project it onto the in-vehicle head-up display to provide an immersive visual experience for the user.

[0031] S5. Receive the user's voice commands, gesture actions, or gaze directions. Specifically, in this step, a hidden Markov model can be used for speech recognition, a fast object detection algorithm can be used for gesture recognition, and the pupillary corneal reflex method can be used for eye tracking to achieve multimodal interaction.

[0032] S6. Control the functions inside the cockpit according to the voice commands, gesture actions, or gaze directions. Specifically, in this step, corresponding cockpit control operations can be performed according to the recognized user interaction intention, such as adjusting the air conditioner, switching music, navigation, etc.

[0033] Step S1 and step S2 continuously monitor the user state, step S3 dynamically adjusts the environment according to the emotion classification results, step S4 provides visual feedback, and steps S5 and S6 realize the user's active control. Through the embodiments of the present invention, real-time response to the user's emotions and needs can be achieved, providing a personalized and intelligent cockpit interaction experience.

[0034] Further, as an implementation manner of the present invention, step S1, obtaining the user's facial expression and voice information, includes: Collect the user's facial images through an in-vehicle camera, and preprocess the facial images to extract facial features; Collect the user's voice signals through an in-vehicle microphone array, and preprocess the voice signals to extract voice features.

[0035] Specifically, collect the user's facial images through an in-vehicle camera. The in-vehicle camera can be installed at positions such as the steering column, dashboard, or rearview mirror to capture the faces of the driver or passengers. The collected facial images can be preprocessed, including but not limited to operations such as image denoising, light compensation, and size normalization, to improve the accuracy of subsequent feature extraction.

[0036] Feature extraction is performed on the preprocessed facial image. Multiple methods can be used for facial feature extraction, such as geometry-based methods, appearance-based methods, or deep learning methods. In some embodiments, a convolutional neural network can be used to extract high-level features of the face, such as the position and shape information of key points like eyebrows, eyes, and mouth. These features can be used for subsequent emotion classification tasks.

[0037] Furthermore, the voice signal of the user is collected through an in-vehicle microphone array. The microphone array can be composed of multiple microphones, distributed at different positions in the cockpit to capture sounds from all directions. The collected voice signal can be preprocessed, including but not limited to operations such as noise suppression, echo cancellation, and signal enhancement, to improve the voice quality.

[0038] Feature extraction is performed on the preprocessed voice signal. Multiple methods can be used for voice feature extraction, such as Mel Frequency Cepstral Coefficients (MFCC), Linear Prediction Coefficients (LPC), or spectrogram. In some embodiments, a deep neural network can be used to directly extract features from the original voice waveform to capture the time-frequency characteristics and semantic information of the voice. These features can be used for subsequent speech recognition and emotion analysis tasks.

[0039] In some embodiments, the extraction of facial features and voice features can be performed in real time to support continuous monitoring and analysis of the user's emotional state. The extracted features can be represented in the form of vectors or matrices for subsequent processing by machine learning algorithms.

[0040] By combining facial expressions and voice information, the emotional state of the user can be captured more comprehensively, improving the accuracy and robustness of emotion recognition. This multi-modal emotion recognition method can adapt to different environmental conditions and individual differences, providing more reliable emotion perception capabilities for the intelligent cockpit.

[0041] Furthermore, as an embodiment of the present invention, in step S2, the use of a deep learning model to perform emotion classification on facial expressions and voice information includes: Using a preset convolutional neural network to perform emotion classification on the facial expression; Using a preset recurrent neural network to perform emotion classification on the voice information.

[0042] Specifically, for the emotion classification of facial expressions, a preset convolutional neural network (CNN) can be used. The CNN can include multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layer is used to extract local features of the facial image, such as edges, textures, etc. The pooling layer is used to reduce the dimension of the feature map and improve the computational efficiency of the model. The fully connected layer is used to map the extracted features to different emotion categories.

[0043] The implementation process of facial expression emotion classification based on convolutional neural network includes: First, the preprocessed facial images are input into the first convolutional layer. The convolutional layer uses multiple filters to perform convolution operations on the input images to generate feature maps. Then, the feature maps undergo non-linear transformation through activation functions (such as ReLU). Next, the pooling layer downsamples the feature maps to reduce the number of parameters. This process is repeated in multiple convolutional layers and pooling layers to gradually extract higher-level features. Finally, the fully connected layer converts the extracted features into a probability distribution of emotion categories.

[0044] In some embodiments, the structure of the CNN may include: 3-5 convolutional layers, each followed by a max pooling layer; 2-3 fully connected layers; and the last layer uses the softmax activation function to output the probability of emotion categories. The specific number of layers and parameters can be adjusted according to the actual application scenario and dataset.

[0045] For the emotion classification of speech information, a preset recurrent neural network (RNN), especially the long short-term memory network (LSTM) or gated recurrent unit (GRU), can be used. These RNN variants can effectively process sequential data and capture the temporal dependencies in speech information.

[0046] The implementation process of speech information emotion classification based on recurrent neural network includes: First, the preprocessed speech feature sequence is input into the RNN. The RNN contains hidden units with recurrent connections that can maintain and update the internal state. At each time step, the RNN receives the input at the current moment and the hidden state at the previous moment, and calculates the new hidden state and output. This process is carried out sequentially along the time series, enabling the network to consider the previous context information.

[0047] In some embodiments, the structure of LSTM or GRU may include: an embedding layer that maps speech features to a low-dimensional space; 2-3 layers of bidirectional LSTM or GRU to capture forward and backward context information; and a fully connected layer that maps the output of LSTM or GRU to emotion categories. The last layer of the network also uses the softmax activation function to output the probability of emotion categories.

[0048] These deep learning models need to be pre-trained on large-scale labeled datasets to learn the general features of facial expressions and speech information. Then, the pre-trained models can be applied to specific emotion classification tasks through transfer learning or fine-tuning.

[0049] In practical applications, the embodiments of the present invention can fuse the emotion classification results of facial expressions and speech information to obtain more comprehensive and accurate emotion recognition results. The fusion method can be simple weighted average or more complex ensemble learning algorithms.

[0050] By using these deep learning models, real-time and accurate recognition of the user's emotional state can be achieved, providing an important basis for subsequent cabin environment adjustment and personalized services.

[0051] Further, as an implementation mode of the present invention, in step S3, adjusting the environment atmosphere in the cabin includes: Adjust the brightness and color of the cabin lighting; and / or Adjust the music type and volume in the cabin; and / or Adjust the air conditioning temperature and wind speed in the cabin.

[0052] Specifically, the cabin lighting brightness and color are automatically adjusted based on the user's recognized emotional state. For example, when the user is detected to be in an excited or happy state, the cabin lighting can be adjusted to a bright and warm tone, such as orange or light pink. When the user is detected to be in a state of fatigue or stress, the lighting can be adjusted to a soft and calm tone, such as light blue or light green. The lighting brightness can be fine-tuned according to the intensity of the emotion to create a suitable atmosphere.

[0053] In some embodiments, a suitable music type can be selected from a preset music library according to the emotion classification result. For example, for positive emotions, you can choose fast-paced pop music or light classical music. For calm or relaxed emotions, you can choose soothing light music or natural sound effects. The volume adjustment can be associated with the intensity of the emotion, and the volume can be adjusted appropriately without affecting driving safety.

[0054] In some embodiments, when it is detected that the user is in a state of tension or anxiety, the cabin temperature can be slightly lowered and the wind speed can be increased to provide a cool and comfortable environment. When it is detected that the user is in a state of fatigue or sleepiness, the cabin temperature can be appropriately increased and the wind speed can be reduced to create a warm and comfortable atmosphere.

[0055] In some embodiments, the adjustment of the ambient atmosphere can be gradual to avoid abrupt changes that may cause discomfort to the user. For example, the change of lighting color and brightness can be completed slowly within 5-10 seconds, the music switching can adopt a fade-in and fade-out effect, and the adjustment of air conditioning temperature and wind speed can be gradually achieved within 1-2 minutes.

[0056] In addition, in some embodiments, the environmental adjustment strategy can be customized according to the user's personal preferences and historical data. For example, the user's habit of manually adjusting environmental parameters under different emotional states can be recorded, and this information can be incorporated into the automatic adjustment algorithm. This method can make the adjustment of the environmental atmosphere more in line with the user's personal needs.

[0057] In some embodiments, the ambient atmosphere adjustment can work in coordination with other cockpit functions. For example, when it is detected that the user is in a fatigued state, in addition to adjusting the environmental parameters, the seat massage function can be triggered or a voice reminder can be provided to advise the user to take a rest. This multi-dimensional adjustment method can more comprehensively respond to the emotional needs of the user and provide a better cockpit experience.

[0058] Through the above method, the intelligent and personalized adjustment of the cockpit environment can be achieved, creating a comfortable atmosphere for the user that matches their emotional state, thereby enhancing the driving experience and safety.

[0059] Further, as an embodiment of the present invention, in step S4, a three-dimensional holographic image is created on the in-vehicle head-up display through a holographic projection device, including: Generating a holographic image using a seven-dimensional holographic projection algorithm; Projecting the holographic image onto the in-vehicle head-up display.

[0060] Specifically, a holographic image is generated using a seven-dimensional holographic projection algorithm. This algorithm can capture images from multiple positions around the scene or theme of the hologram. In some embodiments, multiple cameras or a movable camera can be used to collect image data from different angles and positions. This image data can include three-dimensional spatial information, two-dimensional angular information, time information, and light property information.

[0061] In some embodiments, the algorithm can use Fourier transform to convert the collected image data into the frequency domain, and then reconstruct the three-dimensional light field through digital holography technology. This method can generate high-quality holographic images with depth, view dependence, and dynamic characteristics.

[0062] In some embodiments, a specially designed holographic projection device can be used to project the holographic image onto the transparent screen of the in-vehicle head-up display. This projection device may include a laser light source, a spatial light modulator, and an optical system. The laser light source can provide light with high brightness and high color saturation. The spatial light modulator can modulate the phase or amplitude of the incident light according to the calculated holographic pattern. The optical system can include elements such as lenses and mirrors for precisely projecting the modulated light onto the in-vehicle head-up display.

[0063] In some embodiments, factors such as the vehicle driving state and environmental light need to be considered during the projection process. For example, the brightness and contrast of the holographic image can be dynamically adjusted according to the vehicle speed and road conditions to ensure clear visibility under different driving conditions. In addition, adaptive optical technology can be used to compensate for optical distortions caused by vehicle vibration or temperature changes, ensuring the stability and quality of the holographic image.

[0064] Through the above method, a three-dimensional holographic image with high realism and interactivity can be created on the in-vehicle head-up display. This holographic display technology can provide a more intuitive and immersive information display method for the driver, such as three-dimensional navigation guidance, augmented reality warning prompts, etc., thereby enhancing the driving experience and safety.

[0065] Furthermore, as an implementation manner of the present invention, the seven-dimensional holographic projection algorithm includes: Capturing images from multiple positions; Generating a holographic image by combining three-dimensional position, two-dimensional angle, time, and light attribute information.

[0066] Exemplarily, the seven-dimensional holographic projection algorithm can generate a holographic image by combining the following dimensional information: 1. Three-dimensional space coordinates (X, Y, Z): Describing the position of an object in three-dimensional space; 2. Two-dimensional angles (θ, φ): Describing the viewing angle of the observer relative to the object; 3. Time (t): Used to capture the movement of dynamic scenes or objects; 4. Light attribute (I): Including characteristics such as light intensity, wavelength, polarization, etc.

[0067] The capturing images from multiple positions includes: Using multiple fixed cameras or a movable camera to collect image data from different angles and positions around the target scene or object. In some implementation manners, the camera can move along a preset trajectory to capture a continuous image sequence. The collected image data can contain the three-dimensional space information, viewing angle information, time information, and optical characteristic information of the target.

[0068] Generating a holographic image by combining three-dimensional position, two-dimensional angle, time, and light attribute information. This step can comprehensively process the collected multi-dimensional data to reconstruct the complete light field information. Specifically:

[0069] The three-dimensional position information (X, Y, Z) can be used to describe the position distribution of the target object in space. In some implementation manners, a depth camera or structured light technology can be used to obtain accurate depth information; The two-dimensional angle information (θ, φ) can represent the viewing angle of the observer relative to the target. In some implementation manners, the angle information can be determined by calculating the geometric relationship between different camera positions; The time information (t) can be used to capture the movement of dynamic scenes or objects. In some implementation manners, high-speed imaging technology can be used to record the time evolution process of the target; The light attribute information (I) can include characteristics such as light intensity, wavelength, polarization state, etc. In some implementation manners, devices such as a spectrometer or a polarization filter can be used to measure detailed optical characteristics.

[0070] In some embodiments, the working principle of the seven-dimensional holographic projection algorithm can be based on digital holography technology. First, the collected multi-dimensional image data can be transformed into the frequency domain through Fourier transform. Then, digital holographic reconstruction algorithms, such as the angular spectrum method or the convolution method, can be used to calculate the complex amplitude distribution of the target. Finally, the three-dimensional light field can be reconstructed through inverse Fourier transform.

[0071] In the implementation process, parallel computing technology can be adopted to accelerate data processing. In some embodiments, a graphics processing unit (GPU) or a dedicated holographic computing chip can be used to improve the computing efficiency. In addition, technologies such as compressive sensing can be applied to reduce data storage and transmission requirements.

[0072] The generated holographic image can contain rich depth, perspective, and dynamic information. In some embodiments, the quality and visual effect of the holographic image can be optimized by adjusting the parameters of the reconstruction algorithm, such as the reconstruction distance and the field of view angle.

[0073] Through the above method, the seven-dimensional holographic projection algorithm can generate holographic images with high realism and interactivity, providing a basis for subsequent display and applications.

[0074] Furthermore, as an embodiment of the present invention, step S5, receiving the user's voice command, gesture action, or gaze direction, includes: Using a hidden Markov model for speech recognition; Using a fast object detection algorithm for gesture recognition; Using the pupil-corneal reflection method for eye tracking.

[0075] Specifically, using a hidden Markov model for speech recognition. The hidden Markov model is a statistical model that can be used to model time series data. In speech recognition, the hidden Markov model can regard the speech signal as an observation sequence generated by a series of hidden states.

[0076] In some embodiments, the speech recognition process may include the following steps: 1. Feature extraction: Convert the input speech signal into a sequence of feature vectors, such as Mel-frequency cepstral coefficients (MFCC); 2. Acoustic model: Use a hidden Markov model to establish an acoustic model for each phoneme or phoneme combination. Each model may contain multiple states, the transition probabilities between states, and the probability distribution of generating observations for each state;

[0077] 3. Language model: Establish a transition probability model between words to constrain the grammatical correctness of the recognition result; 4. Decoding: Use the Viterbi algorithm to find the path that is most likely to generate the observation sequence among all possible state sequences, thereby obtaining the recognition result.

[0078] In some embodiments, the Swift-YOLO algorithm can be used for gesture recognition. Swift-YOLO is a variant of the YOLO (You Only Look Once) object detection algorithm, which has a faster detection speed and higher accuracy.

[0079] In some embodiments, the process of gesture recognition may include the following steps: 1. Input image preprocessing: Resize the image and normalize the pixel values; 2. Feature extraction: Use a convolutional neural network to extract multi-scale features of the image; 3. Object detection: Divide the image into grids, and each grid predicts multiple bounding boxes and their confidence levels and class probabilities; 4. Non-maximum suppression: Remove overlapping detection boxes and retain the detection result with the highest confidence; 5. Post-processing: Classify and recognize the detected hand regions according to predefined gesture classes.

[0080] In some embodiments, the pupil corneal reflex method can be used for eye tracking. The pupil corneal reflex method is a non-invasive eye movement tracking technique that can estimate the gaze direction by analyzing the light reflected by the eyes.

[0081] In some embodiments, the process of eye tracking may include the following steps: 1. Light source projection: Emit low-power active near-infrared light towards the human eye. The near-infrared light is harmless to the human eye and does not interfere with normal vision; 2. Image acquisition: Use a high-resolution camera or sensor array to capture the near-infrared image of the eye; 3. Image processing: Preprocess the captured image, such as denoising and contrast enhancement; 4. Feature extraction: Locate the pupil center and the corneal reflection point (i.e., the reflection of the light source on the corneal surface); 5. Gaze estimation: Calculate the rotation angle and gaze direction of the eyeball based on the relative positions of the pupil center and the corneal reflection point; 6. Calibration and mapping: Map the estimated gaze direction to the actual screen coordinates or three-dimensional space coordinates.

[0082] In some embodiments, multiple recognition technologies can be combined. For example, speech recognition, gesture recognition, and eye tracking results can be fused to improve the accuracy and naturalness of interaction. For example, the area of user attention can be determined through eye tracking, and then specific operations can be performed in combination with gesture and voice commands.

[0083] Through the above embodiments, accurate recognition and interpretation of the user's multimodal interaction instructions can be achieved, providing an intuitive and natural human-machine interaction interface for the intelligent cockpit.

[0084] Further, as an embodiment of the present invention, the intelligent cockpit interaction method further includes: Using the support vector machine algorithm to learn the user's interaction behavior; Optimizing the control strategy of the cockpit function according to the learning results.

[0085] Specifically, in some embodiments, the support vector machine (SVM) algorithm can be used to learn the user's interaction behavior. SVM is a supervised learning algorithm that can be used for classification and regression tasks. In the intelligent cockpit scenario, the SVM algorithm can be used to analyze and learn the user's interaction patterns, such as operation frequency, operation sequence, operation time, etc. The working principle of SVM is to find an optimal hyperplane in the high-dimensional feature space to separate data points of different categories. For non-linearly separable data, SVM can use the kernel trick to map the data into a high-dimensional space to achieve linear separability.

[0086] In some embodiments, the application process of the SVM algorithm may include the following steps: First, collect user interaction data, such as touch screen operations, voice commands, gesture actions, etc. Then, preprocess and extract features from these data to construct feature vectors. Next, use the labeled training data set to train the SVM model to learn the patterns and rules of the user's interaction behavior. Finally, use the trained model to classify or predict new interaction behaviors. According to the learning results of SVM, the control strategy of the cockpit function can be optimized. For example, the interface layout can be automatically adjusted according to the user's operation habits, the user's possible next operation can be predicted and the corresponding functions can be prepared in advance, or the cockpit environment parameters can be dynamically adjusted according to the user's preferences. This optimization method based on machine learning can improve the intelligent level and user experience of the cockpit function.

[0087] Further, as an embodiment of the present invention, the intelligent cockpit interaction method further includes: Providing personalized navigation, entertainment, and communication services according to the user's historical interaction data and preferences.

[0088] Specifically, in some embodiments, personalized navigation, entertainment, and communication services can be provided based on the user's historical interaction data and preferences. The implementation of such personalized services may include the following steps: First, collect the user's historical interaction data, such as common navigation destinations, music play records, call frequencies, etc. Then, use machine learning algorithms (such as collaborative filtering or clustering analysis) to analyze this data and extract the user's interests and behavior patterns. Based on the analysis results, personalized service content can be customized for the user. For example, in the navigation service, the most suitable route or points of interest can be recommended according to the user's historical driving routes and preferences. In terms of entertainment services, a personalized playlist can be automatically generated or new music content can be recommended according to the user's music play habits. For communication services, the display order of the contact list can be optimized or intelligent reminder functions can be provided according to the user's call records and contact interaction frequencies.

[0089] In some embodiments, the optimization of personalized services can adopt a continuous learning approach. By real-time recording the user's feedback on the recommended content (such as accepting or rejecting the recommendation), the user preference model can be continuously updated. This dynamic optimization mechanism can ensure that the personalized service can adapt to the changes in the user's interests and provide more accurate and timely services. In addition, context-aware technology can be introduced to consider the user's current environmental factors (such as time, location, weather, etc.) to adjust the service content. For example, the navigation route suggestions or music recommendation strategies can be dynamically adjusted at different times or under different weather conditions. In this way, highly personalized and context-aware intelligent services can be achieved, improving the user's usage experience and satisfaction.

[0090] Referring to Figure 2 , the embodiment of the present invention also discloses an intelligent cockpit interaction system, including an emotion acquisition module 1, an emotion classification module 2, an atmosphere adjustment module 3, a holographic projection module 4, an instruction recognition module 5, a function control module 6, and an optimization training module 7.

[0091] The emotion acquisition module 1 may include an in-vehicle camera and a microphone array. The in-vehicle camera can be installed at positions such as the steering column, dashboard, or rearview mirror for collecting the user's facial images. The microphone array can be composed of multiple microphones distributed at different positions in the cockpit for collecting the user's voice signals. In some embodiments, the emotion acquisition module 1 may further include an image preprocessing unit and an audio preprocessing unit for performing noise reduction, enhancement, etc. on the collected raw data.

[0092] The emotion classification module 2 may include a deep learning processor and a memory. The deep learning processor may be used to run pre-trained convolutional neural networks and recurrent neural networks to perform emotion classification on facial expressions and voice information respectively. The memory may be used to store neural network model parameters and classification results. In some embodiments, the emotion classification module 2 may employ a dedicated AI chip to improve classification speed and energy efficiency.

[0093] The atmosphere adjustment module 3 may include a lighting controller, an audio controller, and an air conditioning controller. These controllers may adjust parameters such as the lighting brightness and color, music type and volume, air conditioning temperature and wind speed, etc. inside the cockpit according to the emotion classification results. In some embodiments, the atmosphere adjustment module 3 may include environmental sensors for real-time monitoring of environmental parameters such as light, temperature, and humidity inside the cockpit to achieve more precise atmosphere adjustment.

[0094] The holographic projection module 4 may include a laser light source, a spatial light modulator, and an optical system. The laser light source may provide light with high brightness and high color saturation. The spatial light modulator may modulate the phase or amplitude of the incident light according to the calculated holographic pattern. The optical system may include elements such as lenses and mirrors for precisely projecting the modulated light onto the in-vehicle head-up display. In some embodiments, the holographic projection module 4 may include adaptive optical elements for compensating for optical distortions caused by vehicle vibration or temperature changes.

[0095] The instruction recognition module 5 may include a voice processor, an image processor, and an eye tracker. The voice processor may run the hidden Markov model algorithm to achieve the recognition of voice instructions. The image processor may run the Swift-YOLO algorithm to achieve the recognition of gesture actions. The eye tracker may use the pupil corneal reflection method to achieve the recognition of the user's gaze direction. In some embodiments, the instruction recognition module 5 may include a multimodal fusion processor for comprehensively analyzing voice, gesture, and gaze information to improve the accuracy of instruction recognition.

[0096] The function control module 6 may include a central processor and a system bus. The central processor may control the execution of various functions inside the cockpit, such as navigation, entertainment, communication, etc., according to the instruction recognition results. The system bus may connect each function module to achieve the transmission of data and control signals. In some embodiments, the function control module 6 may adopt a distributed architecture to allocate some processing tasks to each subsystem to improve the system's response speed and reliability.

[0097] The optimization training module 7 may include a machine learning processor and a large-capacity memory. The machine learning processor may run a support vector machine algorithm to learn and analyze the user's interaction behavior. The large-capacity memory may be used to store the user's historical interaction data and learning results. In some embodiments, the optimization training module 7 may adopt edge computing technology to complete partial data processing and model update tasks locally, reducing the dependence on the cloud server.

[0098] In terms of the system architecture, the intelligent cockpit interaction system may include a hardware layer, a system software layer, a functional software layer, and an application layer. The hardware layer may include the hardware components of each of the above modules, such as processors, memories, sensors, etc. The system software layer may include an operating system, drivers, and middleware for managing hardware resources and providing basic services. The functional software layer may include software implementations of various algorithms and functional modules, such as emotion classification algorithms, holographic projection algorithms, instruction recognition algorithms, etc. The application layer may include various user-oriented application programs, such as navigation applications, entertainment applications, communication applications, etc.

[0099] Through this modular and hierarchical system architecture, the intelligent cockpit interaction system can achieve flexible combination and expansion of functions, while ensuring the stability and maintainability of the system. Each module can communicate and exchange data through standardized interfaces to achieve the overall collaborative work of the system.

[0100] The embodiment of the present invention also discloses a readable storage medium.

[0101] A readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in any one of the above embodiments. The computer-readable storage medium may include: any entity or device capable of carrying the computer program, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), and software distribution media, etc. The computer program includes computer program code. The computer program code may be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), and software distribution media, etc.

[0102] Any process or method description represented in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0103] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing a logical function, and can be embodied specifically in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processing module, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device.

[0104] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A smart cockpit interaction method, characterized in that: include: Obtain user's facial expressions and voice information; Using a deep learning model to perform emotion classification on the facial expressions and voice information; Based on the emotion classification result, adjusting the environment atmosphere in the cockpit; Create a three-dimensional holographic image on the vehicle head-up display through a holographic projection device; Receive user's voice commands, gestures or gaze direction; and controlling functions within the cockpit according to the voice command, gesture action or gaze direction.

2. The intelligent cockpit interaction method according to claim 1, characterized in that: The obtaining of the user's facial expression and voice information includes: Capturing a user's facial image through a vehicle-mounted camera, and preprocessing the facial image to extract facial features; The user's voice signal is collected through the vehicle-mounted microphone array, and the voice signal is preprocessed to extract voice features.

3. The intelligent cockpit interaction method according to claim 1, characterized in that: The method of using a deep learning model to classify facial expressions and voice information includes: Using a preset convolutional neural network to perform emotion classification on the facial expressions; Use a preset recurrent neural network to perform sentiment classification on speech information.

4. The intelligent cockpit interaction method according to claim 1, characterized in that: The adjusting the environment atmosphere in the cockpit includes: Adjust the brightness and color of the cabin lighting; and / or Adjust the music type and volume in the cabin; and / or Adjust the air conditioning temperature and wind speed in the cabin.

5. The intelligent cockpit interaction method according to claim 1, characterized in that: The method of creating a three-dimensional holographic image on a vehicle head-up display by using a holographic projection device includes: Generate holographic images using a seven-dimensional holographic projection algorithm; The holographic image is projected onto a head-up display in a vehicle.

6. The intelligent cockpit interaction method according to claim 6, characterized in that: The seven-dimensional holographic projection algorithm includes: Capture images from multiple locations; A holographic image is generated by combining 3D position, 2D angle, time and light property information.

7. The intelligent cockpit interaction method according to claim 1, characterized in that: The receiving of a user's voice command, gesture action or gaze direction includes: Speech recognition using Hidden Markov Models; Use fast object detection algorithm for gesture recognition; Eye tracking was performed using the pupil-corneal reflection method.

8. The intelligent cockpit interaction method according to claim 1, characterized in that: Also includes: Use support vector machine algorithm to learn users' interactive behaviors; Optimize the control strategy of cockpit functions based on the learning results.

9. The intelligent cockpit interaction method according to claim 9, characterized in that: Also includes: Provide personalized navigation, entertainment and communication services based on users' historical interaction data and preferences.

10. An intelligent cockpit interaction system, characterized in that: The system comprises: An emotion acquisition module, used to acquire at least the user's facial expression and voice information; An emotion classification module, at least used to perform emotion classification on the facial expression and voice information using a deep learning model; An atmosphere adjustment module, at least for adjusting the environment atmosphere in the cabin based on the emotion classification result; A holographic projection module, at least used to create a three-dimensional holographic image on a vehicle head-up display through a holographic projection device; A command recognition module, at least for receiving a user's voice command, gesture action or gaze direction; And, a function control module is at least used to control the functions in the cockpit according to the voice command, gesture action or gaze direction.

11. The intelligent cockpit interaction system according to claim 10, characterized in that: Also includes: The optimization training module is at least used to learn the user's interactive behavior using a support vector machine algorithm, and optimize the control strategy of the cockpit function according to the learning results.

12. A readable storage medium, characterized in that: The readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images