A voice recognition enhancement and multi-modal interaction method for a smart cockpit

Abstract

The application relates to a voice recognition enhancement and multi-modal interaction method for a smart cockpit, which comprises the following steps: acquiring multi-modal data of a driver, performing multi-modal recognition on the multi-modal data, and acquiring voice recognition text and lip movement recognition text of the multi-modal data; taking a first candidate probability in an N-best candidate word list corresponding to the voice recognition text and the lip movement recognition text as global confidence, which is used to define a voice modal weight and a lip movement modal weight; performing basic probability distribution and combination by using the voice modal weight and the lip movement modal weight, acquiring a highest fusion trust degree as a result of voice recognition enhancement; and inputting the result of the voice recognition enhancement into a locally deployed artificial intelligence large model to generate corresponding control instructions or natural language replies. The application solves the problems of insufficient reliability of current smart cockpit voice recognition in a noisy environment and insufficient diversity of interactive functions.

Description

Technical Field

[0001] This invention relates to the fields of intelligent cockpit technology, audio signal processing, image and video detection algorithms, and artificial intelligence, and in particular to a method for enhancing speech recognition and multimodal interaction for intelligent cockpits. Background Technology

[0002] With the advancement of automotive intelligence, intelligent cockpits have become one of the core competitive advantages of modern automobiles. Within the mobile space of the intelligent cockpit, voice interaction, with its natural, convenient, and safe characteristics, is considered the most important human-vehicle interaction method. However, in complex driving environments, voice recognition systems often fail to "hear clearly" or "understand," severely impacting the user experience. This situation has prompted academia and industry to continuously explore voice recognition enhancement technologies, striving to create more intelligent and reliable in-vehicle voice interaction systems.

[0003] Currently, in-vehicle voice recognition systems face multiple challenges. First, the complex acoustic environment is a major factor affecting recognition performance. Background noise such as road noise, wind noise, and engine noise generated during vehicle operation severely interferes with voice signals, while sound wave reflections caused by the confined space inside the vehicle cause reverberation, further reducing voice clarity. Even more problematic is the "cocktail party effect," where when multiple people are conversing simultaneously in the car, the system often struggles to accurately distinguish the target speaker's voice, leading to frequent false wake-ups and recognition errors. Second, the technical bottleneck of far-field recognition cannot be ignored. Due to the distance between the driver and the microphone, the voice signal gradually attenuates during propagation, further reducing the signal-to-noise ratio. Although most smart cockpits are equipped with multi-microphone arrays and beamforming technology to improve sound pickup, their performance remains unstable in dynamic driving scenarios. Furthermore, insufficient semantic understanding capabilities are also a significant factor restricting user experience. Users in the car often use concise, conversational expressions, but existing systems have limited ability to understand such ambiguous commands and lack in-depth multi-turn dialogue management capabilities, making it difficult to understand referential words and implied intentions.

[0004] To overcome these technological bottlenecks, multimodal interaction is considered a crucial solution. However, current multimodal interaction development is still in its early stages and suffers from significant shortcomings. The most prominent problem is the "pseudo-fusion" phenomenon, where many systems simply stack different interaction methods such as voice, touch, and gestures without achieving true semantic-level deep fusion. For example, when a user says "Turn the air conditioning here" and gestures a specific direction, the system often fails to understand the connection between these two modal signals. Simultaneously, the utilization of visual modalities is insufficient. Lip reading, a highly valuable visual cue in noisy environments, has not yet been widely applied in in-vehicle systems due to technological complexity and computational resource limitations. Furthermore, the system's perception and utilization of visual information such as driver status and gaze focus are significantly inadequate, leading to an inability to accurately determine the true intent of voice commands. A deeper problem lies in the general lack of high-level contextual awareness in existing systems. They often fail to comprehensively understand diverse information such as vehicle status, user status, and the external environment, thus hindering adaptive adjustment of interaction strategies. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a speech recognition enhancement and multimodal interaction method for smart cockpits, solving the problems of insufficient reliability of current smart cockpit speech recognition in noisy environments and insufficient diversity of interactive functions.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for enhancing speech recognition and multimodal interaction in smart cockpits includes:

[0008] Acquire multimodal data of the driver, perform multimodal recognition on the multimodal data, and obtain speech recognition text and lip reading text of the multimodal data;

[0009] The first candidate probability in the N-best candidate word list corresponding to the speech recognition text and the lip reading text is used as the global confidence score to define the speech modality weight and the lip reading modality weight.

[0010] The speech modality weights and lip-reading modality weights are used to perform basic probability allocation and combination to obtain the highest fusion trust level as the result of speech recognition enhancement;

[0011] The enhanced speech recognition results are input into a locally deployed large-scale artificial intelligence model to generate corresponding control commands or natural language responses.

[0012] Optionally, obtaining the speech recognition text includes:

[0013] The local deployed Sherpa-NCNN lightweight speech recognition model is used to perform real-time speech-to-text conversion on the speech audio signals in the multimodal data:

[0014] The speech audio signal is converted into acoustic features, the acoustic features are input into the target encoder, and CTC prefix beam search streaming decoder or attention decoder are used to obtain short speech recognition results.

[0015] The short speech recognition result is converted into text to obtain the final text string, which is the speech recognition text.

[0016] Optionally, obtaining the lip-reading text includes:

[0017] The lip detection model is used to convert the video stream data in the multimodal data into a continuous sequence of sliced ​​images; the lip detection model is obtained by training a YOLOv8 model using a first training set;

[0018] The sequence of sliced ​​images is input into the lip recognition model to obtain the lip-reading text; the lip recognition model is obtained by training a 3D convolutional neural network model using a second training set; the second training set includes: a Chinese lip-reading dataset;

[0019] During training, a genetic algorithm is used to re-cluster the initial anchor boxes of the 3D convolutional neural network model, and the activation function is replaced by Leaky-ReLU to enhance gradient backpropagation. At the same time, the learning rate decay strategy and optimizer parameters are adjusted.

[0020] Optionally, the N-best candidate word list corresponding to the speech recognition text and the lip reading text includes:

[0021] ;

[0022] ;

[0023] in, This is the N-best candidate word list for the speech recognition text. This is the N-best candidate word list for the lip-reading text. Candidate words, The original score for the candidate words.

[0024] Optionally, obtaining the candidate probability of the candidate probability includes:

[0025] ;

[0026] ;

[0027] in, This represents the candidate probabilities of the N-best candidate word list corresponding to the speech recognition text. This represents the candidate probabilities of the N-best candidate word list corresponding to the lip-reading text, where T is a temperature parameter used to adjust the smoothness of the probability distribution. This represents the raw probability score for candidate words in speech recognition. , The raw scores for lip-reading candidate words.

[0028] Optionally, the speech modal weights and the lip-reading modal weights are defined as follows:

[0029] ;

[0030] ;

[0031] in, For hyperparameters, It is a function that maps SNR to [0,1]. This is a function that maps the visual quality score VQ to [0,1].

[0032] Optionally, obtaining the highest fusion trust level includes:

[0033] The speech modality weights and lip-reading modality weights are normalized to determine the probability assigned to a single candidate word and the recognition frame;

[0034] The candidate word probabilities and the recognition framework probabilities are combined to obtain a joint basic probability allocation;

[0035] Based on the joint basic probability allocation, the fusion trust level of the candidate words is determined, and the highest fusion trust level is output.

[0036] Optionally, obtaining the joint basic probability allocation includes:

[0037] ;

[0038] Where K is the normalization constant, Let B be the set of candidate words supported by the hypothesis or proposition, A be the set of candidate words supported by phonological evidence, and C be the set of candidate words supported by lip-reading evidence. Assign a confidence level to set B for the voice evidence. Assign a confidence level to set C for lip-reading evidence.

[0039] Optionally, determining the fusion trust level of the candidate words includes:

[0040] ;

[0041] in, Both voice and lip reading are explicitly supported. For speech that is explicitly supported but lip reading is uncertain, Lip reading provides explicit support for speech expression that is uncertain.

[0042] The beneficial effects of this invention are as follows:

[0043] This invention introduces lip-reading recognition into the intelligent cockpit's voice recognition function. It extracts visual features of lip movement sequences using a convolutional neural network and fuses them with audio features at the decision layer, thereby enhancing voice recognition. This effectively overcomes the interference of environmental noise on the voice signal, reduces the attenuation of the voice signal during propagation in dynamic driving scenarios, and accurately distinguishes the voice of the target speaker when multiple people are talking simultaneously in the vehicle, reducing false wake-ups and recognition errors, and significantly improving the anti-interference capability of voice recognition. Furthermore, by deploying a lightweight large model locally, the enhanced voice recognition results are interacted with the large model in a multimodal manner. Utilizing the large model's natural language processing technology, the semantic understanding and contextual association of the voice recognition are strengthened. The interaction results are presented through the intelligent cockpit screen or voice commands, and corresponding instructions are executed, further enriching the intelligent functions and human-computer interaction level of the intelligent cockpit, improving the driving experience and safety, and providing users with a richer and more convenient interactive experience. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of a speech recognition enhancement and multimodal interaction method for smart cockpits according to an embodiment of the present invention;

[0046] Figure 2 This is a design architecture diagram of a speech recognition enhancement and multimodal interaction method for smart cockpits according to an embodiment of the present invention;

[0047] Figure 3 This is a structural diagram of a Sherpa-NCNN-based speech recognition system for smart cockpits, according to an embodiment of the present invention.

[0048] Figure 4 This is a structural diagram of a lip-reading recognition system for smart cockpits according to an embodiment of the present invention;

[0049] Figure 5This is a detailed flowchart of the multimodal decision-level fusion algorithm according to an embodiment of the present invention;

[0050] Figure 6 This is a deployment connection diagram of the speech recognition enhancement and multimodal interaction method for smart cockpits according to an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown in the figure, this embodiment discloses a speech recognition enhancement and multimodal interaction method for smart cockpits, including: acquiring multimodal data of the driver; performing multimodal recognition on the multimodal data; acquiring speech recognition text and lip reading text of the multimodal data; using the first candidate probability in the N-best candidate word list corresponding to the speech recognition text and lip reading text as the global confidence level, which is used to define the speech modality weight and lip reading modality weight; using the speech modality weight and lip reading modality weight to perform basic probability allocation and combination, and obtaining the highest fusion confidence level as the result of speech recognition enhancement; inputting the result of speech recognition enhancement into a locally deployed artificial intelligence large model to generate corresponding control commands or natural language responses.

[0054] Specifically, this embodiment discloses a method for enhancing voice recognition and multimodal interaction in smart cockpits, including:

[0055] S1. Deploy microphone and camera devices in the smart cockpit system to acquire multimodal data of the driver's real-time voice and lip movements: Based on the existing smart cockpit in-vehicle system and in-vehicle control terminal, under the control of the controller, deploy a microphone array to collect the user's voice audio signals, and simultaneously deploy high-definition cameras facing the driver and passengers to collect video including the user's lip area, collecting both auditory and visual modal data in noisy environments. The central processing unit in the cockpit is responsible for coordinating the work of the sensors and receiving the audio and video data.

[0056] S2. Upload the voice and lip movement videos collected by the sensor devices to the in-vehicle platform of the smart cockpit. The local algorithm model performs real-time recognition and converts the data into text for storage. The driver's or user's voice commands and lip movement videos are uploaded to the in-vehicle platform in real time. The platform uses a locally deployed Sherpa-ncnn lightweight speech recognition model to convert the voice signals into text in real time, outputting the recognized text and the confidence level of the recognition result. Based on the YOLOv8 object detection model, the uploaded lip movement videos are used for face and lip region detection and tracking, extracting continuous lip movement image sequences. These lip image sequences are then input into a lip-reading recognition model based on a 3D convolutional neural network. The lip image sequences are recognized in real time and converted into text, outputting the recognized text and the confidence level of the recognition result. The results of both modal recognitions and related data are saved locally.

[0057] S3. The system fuses the recognized speech and lip-reading text results using a multimodal decision-level algorithm to enhance speech recognition functionality: Based on the saved results and relevant data, the recognition results of the two modalities are fused, and a weighted, confidence-based fusion strategy is designed. The system establishes confidence assessment mechanisms for the two independent channels of speech recognition and lip-reading recognition, and uses a Bayesian inference fusion algorithm for dynamic confidence weighting. Using DS evidence theory, the two modalities are treated as independent evidence sources, and their output N-best candidate list probability distribution is used as the basic probability allocation. When one modality completely fails due to environmental interference, its "uncertainty" evidence will be dominated by the strong evidence from the other modality, thus leading to a reliable conclusion and enhancing speech recognition.

[0058] S4. Deploy a large-scale AI model locally within the smart cockpit vehicle platform. Input the enhanced speech recognition results into the large model and present the interactive results or execute corresponding commands on the smart cockpit screen. Deploy a lightweight large model locally within the smart cockpit vehicle platform to ensure real-time interaction, offline availability, and data privacy and security. Input the final recognized text after speech recognition enhancement into the locally deployed large model using an efficient TCP Socket inter-process communication mechanism. The large model performs deep semantic understanding and contextual reasoning on user commands to generate corresponding control commands or natural language responses. The output of the large model is sent to the cockpit domain controller. If the output is a control command, the controller executes the corresponding vehicle function; if the output is an information response, the result is presented on the cockpit screen or voice broadcast system, thereby achieving highly intelligent multimodal interaction that surpasses traditional command sets.

[0059] S1. Deploy microphone and camera devices in the smart cockpit system to acquire multimodal data of the driver's real-time voice and lip movements:

[0060] S101. Based on the existing smart cockpit in-vehicle system and in-vehicle control terminal, a microphone array is deployed to collect the user's voice audio signals. Simultaneously, high-definition cameras facing the driver and passengers are deployed to collect video including the user's lip area, acquiring both auditory and visual modal data in noisy environments. The microphone array is typically embedded in the cockpit in a distributed layout, mainly installed in key locations such as the headliner control panel (covering the front driver and passenger seats) and the A-pillar (enhancing directional sound pickup for the driver). The high-definition cameras are mainly deployed on the inside of the A-pillar or above the dashboard, directly facing the driver's lip area, ensuring stable and clear visual signals captured in various seating positions. This distributed layout, through the microphone array's sound source localization and beamforming technology, can accurately locate the target speaker, effectively suppressing environmental noise and interference from other occupants' conversations, significantly improving the signal-to-noise ratio. Furthermore, the cameras are deployed in rigid, fixed positions, ensuring stable lip image sequences even during vehicle vibrations. This collaborative layout constitutes a reliable multimodal data acquisition network, providing a high-quality data foundation for subsequent voice enhancement and lip reading recognition, and is a key physical guarantee for achieving highly robust intelligent interaction. The central processing unit inside the cockpit is responsible for coordinating the operation of the sensors and receiving the audio and video data.

[0061] S102. Overall design framework diagram of the voice recognition enhancement and multimodal interaction system for the smart cockpit, as shown below. Figure 2 As shown. The system mainly consists of a physical hardware layer, a device driver layer, an operating system layer, a functional module layer, and an interactive application layer. The entire system is developed using the Python language. The specific functions of each part of the system are as follows:

[0062] The physical hardware layer forms the physical foundation of the entire system, with its core comprising Huawei Ascend Atlas series boards, microphones, and high-definition cameras. As an in-vehicle edge computing node, the Huawei Ascend Atlas board provides powerful local computing power, ensuring high real-time performance and low latency in data processing, while protecting user privacy data from leaving the vehicle. The microphone and camera act as the system's "sensory organs," respectively responsible for collecting the user's audio and lip-sync video signals. These raw signals are transmitted to the board in real time via its integrated physical USB interface, providing the most basic data source for subsequent recognition and processing. The design and selection of this layer directly determines the quality of the system's data acquisition and the overall system performance ceiling.

[0063] The device driver layer is a crucial interface ensuring that upper-layer software can seamlessly access and control the underlying hardware. It primarily consists of a series of dedicated drivers, including a microphone driver responsible for audio signal acquisition and pre-processing, a camera driver for image capture and initial correction, and a dedicated driver for the Neural Processing Unit (NPU) on the Ascend board. These drivers encapsulate complex hardware operation instructions into a unified API interface that can be called by the operating system. This allows the microphone and camera to be accessed and controlled by the system, and ensures that models such as Sherpa-NCNN and YOLOv8 can efficiently utilize the NPU for inference computation, thereby achieving the abstraction and standardized management of hardware resources.

[0064] The operating system layer is the core layer of the device, responsible for managing and controlling hardware resources, providing hardware abstraction and encapsulating system services for the stable operation of algorithms and model programs. The Huawei Ascend Atlas 200DK developer board uses the classic Ubuntu operating system as its core. This layer undertakes core tasks such as resource scheduling, process management, memory control, and file system services for the entire system. The Ubuntu operating system provides a stable and reliable operating environment for the various functional modules above, abstracting the computing power of heterogeneous hardware (CPU, NPU, etc.) into unified system services. This allows algorithm engineers to focus on model and application development without delving into the specific details of the underlying hardware, greatly improving development efficiency and system maintainability.

[0065] The aforementioned functional module layer is a key component for realizing the enhanced voice recognition and multimodal interaction functions of the smart cockpit. It is the algorithmic core of the entire system's intelligent functions, integrating multiple advanced AI models and processing modules. Specifically:

[0066] 1) Sherpa-NCNN speech recognition model: a highly optimized streaming speech recognition framework responsible for real-time, low-latency text transcription of audio signals.

[0067] 2) YOLOv8 object detection model: responsible for quickly and accurately detecting and locating faces, especially the lip region, from the video stream, providing accurate input for subsequent lip reading.

[0068] 3) Lip reading recognition model based on convolutional neural network: Visual features are extracted from the lip region sequence truncated by YOLOv8 to infer the corresponding pronunciation content.

[0069] 4) Multimodal decision-level fusion algorithm: This is one of the key innovations of this patent. The algorithm receives recognition results (usually text or probability distributions) from two channels: speech recognition and lip reading, and performs adaptive weighted fusion at the decision level. It can automatically assign higher weights to lip reading features in high-noise environments and vice versa, thereby generating a final recognition result that is far more reliable and accurate than that of a single modality.

[0070] 5) DeepSeek Lightweight Local Model: A pruned and optimized advanced large language model deployed locally. It receives fused and enhanced text recognition data, performs deep semantic understanding, contextual association, and dialogue management, enabling truly intelligent interaction.

[0071] 6) TCP Inter-Process Communication Module: Responsible for coordinating data flow and instruction synchronization between the above functional modules to ensure the efficient and stable operation of the entire processing pipeline.

[0072] The interactive application layer is the ultimate embodiment of the system's functional value and serves as the interface through which users directly interact with the smart cockpit system. In this system, this layer primarily provides two core services: first, real-time voice control, allowing users to accurately and reliably control vehicle functions such as navigation, music, and air conditioning through enhanced voice commands; second, multimodal interaction with the local DeepSeek large model, enabling users to perform complex tasks such as open-domain natural dialogue, knowledge-based question answering, and information retrieval. System feedback is clearly presented to the user on the screen as structured information (such as cards and lists) or through direct command execution.

[0073] S2. Upload the voice and lip movement videos collected by the sensor devices to the in-vehicle platform of the smart cockpit, where they are recognized in real time using a local algorithm model and converted into text for storage.

[0074] The S201 uses a built-in microphone array to collect raw user speech data in real time, while simultaneously capturing video streams including lip movements using a high-definition camera. Both raw data streams are stably transmitted to the Ascend Atlas board via USB interface and corresponding device drivers. Within the board, the data is processed in parallel: the speech stream is fed into a locally deployed Sherpa-NCNN lightweight speech recognition model for real-time speech-to-text conversion, outputting the recognized text and its confidence score; the video stream is first processed by a YOLOv8 object detection model for accurate face and lip region detection and tracking, extracting a continuous sequence of lip movement images. This sequence is then input into a lip-reading recognition model based on a 3D convolutional neural network, converting it into corresponding text results and their confidence scores in real time. The system associates and encapsulates the text content and confidence score data generated from the speech and lip-reading modalities respectively, storing them locally to provide standardized structured data for subsequent decision-level fusion.

[0075] The Sherpa-NCNN is a real-time speech recognition and speech activity detection (VAD) framework based on next-generation Kaldi and ncnn. The overall structure of the speech recognition system is as follows: Figure 3 As shown. The main functions of the system include real-time speech recognition and voice activity detection (VAD). The speech recognition process of Sherpa-ncnn typically includes the following steps: 1) Audio acquisition: Reading the audio stream from the microphone in real time. 2) Feature extraction: Converting the raw PCM audio signal into acoustic features (usually FBank features). 3) Encoder inference: Feeding the acoustic features into a Paraformer / Zipformer encoder to extract high-level feature representations. 4) Decoding: Using methods such as CTC prefix beam search for fast, low-latency streaming decoding, or using a more accurate attention decoder to obtain the final result (for short speech recognition). 5) Text output: Converting the decoded tokenID sequence into the final text string. It supports streaming speech-to-text, that is, recognition while speaking, detecting voice activity, and is suitable for various real-time applications.

[0076] The overall structure and workflow of the intelligent cockpit lip-reading system, such as... Figure 4 As shown. The object detection model is developed and trained based on the YOLOv8 model. YOLOv8 supports a wide range of computer vision tasks, including object detection, instance segmentation, pose / keypoint detection, rotated bounding box detection, and classification. Each model variant is optimized for its specific task and is compatible with various operating modes (such as inference, validation, training, and export). Specific operating methods include configuring a Python environment based on Anaconda on a PC and using the NVIDIA CUDA architecture for GPU-accelerated training. This invention collects and constructs a lip image dataset containing tens of thousands of labeled samples, covering lip images of different ethnicities, lighting conditions, head poses, and occlusion scenarios. Through in-depth analysis of the YOLOv8 model structure, this invention performs systematic optimization and debugging: adjusting the input image size to 640x640 to balance accuracy and speed; using a genetic algorithm to re-cluster the initial anchor boxes to better fit lip features; replacing the activation function from SILU to Leaky-ReLU to enhance gradient backpropagation; and carefully adjusting the learning rate decay strategy and optimizer parameters. After hundreds of epochs of iterative training, the final lip detection model is able to perform face detection and lip region localization in real time and accurately under complex in-vehicle environments, varying lighting conditions, and large-angle deflection. The high-quality lip region image sequence output provides a reliable data foundation for subsequent lip reading tasks.

[0077] The lip-reading recognition module is based on the lip-reading-deeplearning-master 3D convolutional neural network model, and has been comprehensively customized and optimized for the actual application scenarios of smart cockpits. In terms of technology selection, the model is built on the TensorFlow framework, ensuring development efficiency and operational stability. To improve the accuracy of Chinese lip-reading recognition, this project abandoned the general English dataset and instead used the LRW1000 Chinese lip-reading dataset, which is more consistent with the characteristics of the Chinese language, for training. Given the significant differences between this dataset and the original model framework in terms of design goals, this invention implemented the following key improvements to the model: First, a fundamental reconstruction was carried out at the model structure level, transforming the original contrastive learning method into a more efficient temporal image classification task, enabling the model to better understand and utilize the continuous changes in lip movements over time. In the data preprocessing stage, the model input was changed from raw video stream data to a continuous sequence of sliced ​​images, and recognition was achieved by classifying these temporal images. Simultaneously, the input image format was changed from the proprietary .npy format to the general jpg format, enhancing the model's adaptability and deployment flexibility across different hardware configurations and environments. In terms of model training strategy, the task objective was redesigned, shifting from contrastive learning to a direct classification task. Accordingly, the training label data format was modified to conform to the annotation specifications of the LRW1000 dataset. Furthermore, the detection regions of the input images were precisely located and adjusted, and a grayscale processing mechanism was introduced. This reduced data dimensionality, allowing the model to focus more on extracting and learning lip movement features, effectively improving the model's recognition performance.

[0078] S3. The text results of the recognized speech and lip reading are fused using a multimodal decision-level algorithm to enhance the speech recognition function:

[0079] S301. Based on the saved results and related data, the recognition results of the two modalities are fused, and a weighted fusion strategy based on confidence is designed. The system establishes confidence evaluation mechanisms for the two independent channels of speech recognition and lip reading, and uses a Bayesian inference fusion algorithm to dynamically weight the confidence. Using DS evidence theory, the two modalities are regarded as independent sources of evidence, and their output N-best candidate list probability distribution is used as the basic probability allocation. When one modality completely fails due to environmental interference, its "uncertainty" evidence will be dominated by the strong evidence of the other modality, thus drawing a reliable conclusion and enhancing speech recognition.

[0080] The process of the decision-level fusion algorithm is as follows: Figure 5 As shown, the specific method is as follows:

[0081] 1) Modal Independent Processing and Confidence Generation: In the speech recognition channel, after inputting the audio signal, the system will output an N-best candidate word list. ,in (i=1,2,……,N) are candidate words, (i=1,2,...,N) represent their raw scores (e.g., log-likelihood scores). Confidence scores are then calculated, converting the raw scores into probability form. Normalization is typically performed using the Softmax function. Where T is a temperature parameter used to adjust the smoothness of the probability distribution. The global confidence score of the entire speech recognition result. The probability of using Top-1 candidates Or it can be measured by entropy.

[0082] In the lip-reading channel, after inputting a video sequence of the lip region, the system outputs an N-best candidate word list. And the confidence level of the original score is calculated using Softmax. Finally, the global confidence score of the entire lip-reading recognition result is obtained. .

[0083] 2) Environmental Perception and Modal Weight Calculation: An environmental perception module is designed to estimate the signal-to-noise ratio (SNR) of the audio signal and calculate the SNR of the input audio in real time. A low SNR indicates an unreliable speech signal. Simultaneously, visual quality assessment is performed by extracting metrics from the video stream, such as the overall brightness and contrast of the lip area, motion blur, and whether it is occluded, to obtain a comprehensive visual quality score (VQ) (range 0-1). A low VQ indicates an unreliable lip-reading signal.

[0084] Subsequently, dynamic weights are calculated based on Bayesian inference principles, and the global confidence score is... and By treating environmental factors SNR and VQ as prior reliability, their weights are dynamically adjusted to reflect new evidence.

[0085] Define the weights of speech modalities Weights of lip-reading modal :

[0086] ;

[0087] ;

[0088] Here, λ is a hyperparameter (e.g., 0.7) used to balance the model’s own confidence with environmental evidence, f(SNR) is a function that maps SNR to [0,1], such as a sigmoid function that is close to 1 when SNR is high and close to 0 when SNR is low, and g(VQ) is similar, mapping the visual quality score VQ to [0,1].

[0089] The weights are normalized to ensure their sum is 1, making them easier to use as weighting factors later. , .

[0090] 3) Constructing the basic probability allocation of DS evidence theory: Defining the identification framework of DS evidence theory. It contains all possible candidate words. To simplify the calculation, the union of the two N-best lists is taken as... Construct a basic probability assignment function to build a basic probability assignment for each mode. and The key step in probability allocation lies in introducing uncertainty. The system allocates only a portion of the probability to the entire recognition framework. This indicates the "uncertainty" of the modality. Assigned to a single candidate word. The probability is:

[0091] ;

[0092] ;

[0093] Note: If If it is not in the N-best list for a certain modality, then its or It is 0.

[0094] Assignment to uncertainty (i.e., framework) The probability of () is:

[0095] ;

[0096] ;

[0097] The design is ingenious: the lower the weight of a modality ( Small), which is assigned to the mass m of uncertainty. The larger the value, the less reliable the source of evidence.

[0098] 4) Dempster Evidence Combination and Decision Making: When applying Dempster's combination rules for evidence fusion, the system assigns the basic probability (BPA) to both speech and lip-reading modalities. and By combining them, we obtain the joint BPA. Its core calculation formula is:

[0099] ;

[0100] The normalization constant K is used to handle conflicts between pieces of evidence, and its calculation formula is as follows:

[0101] ;

[0102] Specifically, the confidence level calculation for a single candidate word w depends on the combination of evidence supporting that candidate word, which comes from three aspects:

[0103] 1. Both speech and lip reading clearly support this word: 2. Voice input is clearly supported, but lip reading is uncertain: 3. Vocal representation is uncertain, while lip reading provides clear support: ;

[0104] Therefore, the fusion confidence level of candidate word w is:

[0105] ;

[0106] In the final decision-making phase, the system calculates the joint support of each candidate word (from the union of the two modal N-best lists). By comparing the fusion confidence of all candidate words, the one with the highest confidence level is selected. The candidate words with the highest values ​​are used as the final identification results of the system. This decision-making mechanism ensures that the most reliable identification conclusion is reached by comprehensively considering the strength of evidence and uncertainty of both modalities.

[0107] S4. Deploy a large-scale artificial intelligence model locally within the smart cockpit in-vehicle platform, input the enhanced voice recognition results into the large-scale model, and display the interactive results or execute corresponding commands on the smart cockpit screen:

[0108] like Figure 6As shown, a lightweight, large-scale language model, DeepSeek-r1:7b, is deployed locally within the vehicle platform. This local deployment improves the real-time performance of interactions, offline availability, and data privacy and security. The final recognized text, enhanced by speech recognition, is input to the locally deployed large model via an efficient TCP Socket inter-process communication mechanism. The large model performs deep semantic understanding and contextual reasoning on user commands, generating corresponding control commands or natural language responses. The output of the large model is sent to the cockpit domain controller. If the output is a control command (such as "lower the air conditioning temperature to 22 degrees"), the controller executes the corresponding vehicle function; if the output is an information response (such as answering the user's question "What are some highly-rated restaurants nearby?"), the result is presented on the cockpit screen or voice broadcast system, thus achieving highly intelligent multimodal interaction that surpasses traditional command sets.

[0109] The specific steps for local deployment of large DeepSeek models are as follows:

[0110] 1) Connect the Huawei Ascend Atlas development board to your computer via USB cable. Open your computer's Device Manager. The system will usually automatically recognize the development board and attempt to install the driver. You will need to manually update the driver and select "Network adapter" under Hardware type, ensuring that the RNDIS (Remote Network Driver Interface Specification) driver is installed. The RNDIS driver allows you to simulate a network device via USB connection, enabling the development board to be recognized by the system as a network adapter. After successful installation, the corresponding network device entry should appear in Device Manager, indicating that the driver is ready.

[0111] 2) Configure network settings. On the PC, go to "Network and Internet settings" and find the "Change adapter options" interface. The "Local Area Connection" corresponds to the Ascend development board's network interface. Right-click on this connection, enter the properties window, select "Internet Protocol Version 4 (TCP / IPv4)," and click the Properties button. Here, you need to manually set the IP address and subnet mask. Set the IP address to 192.168.0.1 and the subnet mask to 255.255.255.0.

[0112] 3) Log in to the developer tools. Use MobaXterm, a multi-functional remote terminal software. After opening the software, go to the Session settings interface, select the SSH connection method, and enter the previously set development board IP address (e.g., 192.168.1.2) in the Remote Host field. Keep the port as the default 22, and enter the developer name (which can be set arbitrarily) in the Username field. After a successful connection, the user will enter the development board's built-in Ubuntu system command-line interface. For graphical operation, you can use the VNC protocol to connect to the same IP address to access the development board's visual desktop environment.

[0113] 4) Install the DeepSeek model. First, install Ollam using the official script, then use a specific pull command to download the DeepSeek model. For example, running `ollam run deepseek-r1:7b` will launch the 7B parameter version of the DeepSeek model. After the model has finished loading, the development board will enter AI question-and-answer mode, capable of thinking about and answering user-input questions.

[0114] 5) Optimize the model for inference speed by switching to the 1.5B version model with fewer parameters to reduce computational load; install necessary drivers to fully utilize the development board's built-in NPU (Neural Processing Unit) and optimize model running efficiency through a dedicated AI acceleration library. The NPU can efficiently perform typical AI tasks such as matrix operations, significantly improving the model's inference speed and making the locally deployed AI assistant more practical.

[0115] The specific process for establishing TCP inter-process communication between the speech recognition enhancement result and the DeepSeek large model is as follows:

[0116] 1) Server-side initialization: When the DeepSeek server starts, it first calls socket.socket(socket.AF_INET, socket.SOCK_STREAM) to create a TCP socket, specifying the use of the IPv4 address family and connection-oriented streaming. Then it calls bind(('localhost', 8888)) to bind the socket to port 8888 of the local loopback address, and then executes listen(5) to start the listening state, setting the maximum number of waiting connections to 5. At this time, the server enters a blocking waiting state. socket.socket() is the socket creation function; socket.AF_INET is the address family, specifying the use of the IPv4 protocol; socket.SOCK_STREAM is the socket type, specifying the use of connection-oriented streaming, i.e., the TCP protocol. `bind()` is the binding function that associates a socket with a specific network address and port number; 'localhost' is the host address, here referring to the local loopback address (127.0.0.1); 8888 is the port number; `listen()` is the listening function that puts the socket into a passive connection-accepting state.

[0117] 2) Client Connection Initiation: After the Sherpa-ncnn client starts, it also creates a TCP socket and then calls the connection function connect(('localhost', 8888)) to actively initiate a connection request to the server. The system performs a TCP three-way handshake: the client sends a synchronization message, the server returns an acknowledgment message, and the client finally sends an ACK acknowledgment message to complete the handshake.

[0118] 3) Server accepts connection: When the server detects a connection request using the accept() function, the kernel retrieves a connection from the completed connection queue, creates a brand new socket for that client for dedicated communication, and the original listening socket continues to wait for other connections. The server records the client's address information and starts a dedicated thread to handle the connection.

[0119] 4) Data Transmission Channel Establishment: After the connection is established, a bidirectional communication channel is formed between the two parties. The client serializes the speech-recognized text into JSON format using the send() function and sends it. The server receives and parses the data using the recv() function. After processing, the server also returns response data through the socket. The entire process maintains an unbroken connection and supports multiple request-response interactions.

[0120] 5) Connection Maintenance and Termination: During communication, both parties maintain a close connection through a heartbeat mechanism. When the client exits or times out, it actively closes the connection using `close()`, triggering the TCP four-way handshake process and releasing port resources. The server then cleans up thread resources accordingly and prepares to accept new connections.

[0121] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for enhancing speech recognition and multimodal interaction in smart cockpits, characterized in that, include: Acquire multimodal data of the driver, perform multimodal recognition on the multimodal data, and obtain speech recognition text and lip reading text of the multimodal data; The candidate probabilities in the N-best candidate word lists corresponding to the speech recognition text and the lip reading text are used as global confidence scores to define the speech modality weights and lip reading modality weights. Obtaining the candidate probabilities includes: ； ； in, This represents the candidate probabilities of the N-best candidate word list corresponding to the speech recognition text. This represents the candidate probabilities of the N-best candidate word list corresponding to the lip-reading text, where T is a temperature parameter used to adjust the smoothness of the probability distribution. , This represents the raw probability score for candidate words in speech recognition. , The original score for lip-reading candidate words; The speech modality weights and lip-reading modality weights are used to perform basic probability allocation and combination to obtain the highest fusion trust level as the result of speech recognition enhancement; The enhanced speech recognition results are input into a locally deployed large-scale artificial intelligence model to generate corresponding control commands or natural language responses.

2. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 1, characterized in that, Obtaining the speech recognition text includes: The local deployed Sherpa-NCNN lightweight speech recognition model is used to perform real-time speech-to-text conversion on the speech audio signals in the multimodal data: The speech audio signal is converted into acoustic features, the acoustic features are input into the target encoder, and CTC prefix beam search streaming decoder or attention decoder are used to obtain short speech recognition results. The short speech recognition result is converted into text to obtain the final text string, which is the speech recognition text.

3. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 1, characterized in that, Obtaining the lip-reading text includes: The video stream data in the multimodal data is converted into a continuous sequence of sliced ​​images using a lip detection model; the lip detection model is obtained by training a YOLOv8 model using a first training set. The sequence of sliced ​​images is input into the lip recognition model to obtain the lip-reading text; the lip recognition model is obtained by training a 3D convolutional neural network model using a second training set; the second training set includes: a Chinese lip-reading dataset; During training, a genetic algorithm is used to re-cluster the initial anchor boxes of the 3D convolutional neural network model, and the activation function is replaced by Leaky-ReLU to enhance gradient backpropagation. At the same time, the learning rate decay strategy and optimizer parameters are adjusted.

4. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 1, characterized in that, The N-best candidate word list corresponding to the speech recognition text and the lip reading text includes: ； ； in, This is the N-best candidate word list for the speech recognition text. This is the N-best candidate word list for the lip-reading text. Candidate words, The original score for the candidate words.

5. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 1, characterized in that, The speech modal weights and the lip-reading modal weights are defined as follows: ； ； in, For hyperparameters, It is a function that maps SNR to [0,1]. The function that maps the visual quality score VQ to [0,1]. This represents the global confidence score for the entire speech recognition result. This represents the global confidence level of the entire lip-reading result.

6. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 1, characterized in that, Obtaining the highest level of fusion trust includes: The speech modality weights and lip-reading modality weights are normalized to determine the probability assigned to a single candidate word and the recognition frame; The candidate word probabilities and the recognition framework probabilities are combined to obtain a joint basic probability allocation; Based on the joint basic probability allocation, the fusion trust level of the candidate words is determined, and the highest fusion trust level is output.

7. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 6, characterized in that, Obtaining the combined basic probability allocation includes: ； Where K is the normalization constant, Let B be the set of candidate words supported by the hypothesis or proposition, A be the set of candidate words supported by phonological evidence, and C be the set of candidate words supported by lip-reading evidence. Assign a confidence level to set B for the voice evidence. Assign a confidence level to set C for lip-reading evidence.

8. The speech recognition enhancement and multimodal interaction method for smart cockpits according to claim 6, characterized in that, Determining the fusion trust level of the candidate words includes: ； in, Both voice and lip reading are explicitly supported. For speech that is explicitly supported but lip reading is uncertain, Lip reading provides explicit support for speech expression that is uncertain.

Images